Nov 02 12:24:06.241708 ip-10-0-135-160 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 12:24:06.241720 ip-10-0-135-160 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 12:24:06.241729 ip-10-0-135-160 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 12:24:06.242015 ip-10-0-135-160 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 12:24:16.425382 ip-10-0-135-160 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 12:24:16.425401 ip-10-0-135-160 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 00570ddcd6f1459f88649a976f362aa4 --
Nov 02 12:26:00.858155 ip-10-0-135-160 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:26:01.312622 ip-10-0-135-160 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.312622 ip-10-0-135-160 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:26:01.312622 ip-10-0-135-160 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.312622 ip-10-0-135-160 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.319101 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319010    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.319101 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319097    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.319101 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319106    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319111    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319116    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319120    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319125    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319129    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319133    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319137    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319141    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319145    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319149    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319154    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319158    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319163    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319167    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319171    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319175    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319180    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319184    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.319319 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319188    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319193    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319197    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319201    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319205    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319210    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319225    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319231    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319236    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319241    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319246    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319250    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319256    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319261    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319266    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319271    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319276    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319281    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319286    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319290    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.320126 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319295    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319300    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319304    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319309    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319314    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319318    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319323    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319327    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319332    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319337    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319342    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319346    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319351    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319356    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319361    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319366    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319371    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319377    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319382    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.320656 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319387    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319391    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319396    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319401    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319405    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319410    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319415    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319420    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319427    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319432    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319436    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319441    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319446    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319450    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319457    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319463    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319469    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319474    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319479    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.321370 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319484    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319488    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319493    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319498    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319513    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319518    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319523    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319528    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319533    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319538    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319543    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319548    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319552    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319557    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319562    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319566    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319571    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319576    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319582    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319587    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.322265 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319591    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319597    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319601    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319607    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319611    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319616    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.319620    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.321985    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.321997    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322002    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322008    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322012    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322017    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322022    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322027    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322032    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322036    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322050    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322055    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322060    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.323141 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322066    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322071    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322075    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322080    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322085    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322089    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322094    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322098    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322103    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322107    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322112    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322116    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322121    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322129    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322137    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322143    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322149    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322155    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322160    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.324048 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322165    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322171    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322176    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322183    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322189    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322194    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322199    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322204    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322210    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322215    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322219    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322232    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322238    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322243    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322248    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322252    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322257    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322262    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322266    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.324616 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322270    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322275    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322279    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322284    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322288    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322292    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322297    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322302    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322307    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322312    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322317    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322321    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322326    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322331    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322335    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322340    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322344    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322349    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322355    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.325100 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322359    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322364    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322369    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322373    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322378    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322383    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322396    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322402    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322406    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322411    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322415    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322420    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322424    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322429    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322434    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322439    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322443    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322448    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322452    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322457    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.325808 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322462    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322466    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322471    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322478    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322483    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322488    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322493    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322497    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322502    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322506    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322511    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322515    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322520    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322525    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322530    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.322535    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323532    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323544    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323559    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.326684 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323565    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323570    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323576    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323583    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323589    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323595    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323600    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323606    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323611    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323616    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323620    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323625    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323630    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323635    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323640    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323645    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323650    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323654    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323659    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.327515 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323664    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323670    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323674    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323679    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323684    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323688    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323693    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323697    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323703    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323707    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323713    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323718    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323722    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323736    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323741    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323749    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323753    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323757    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323770    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323775    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.328186 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323780    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323785    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323789    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323794    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323799    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323803    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323807    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323812    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323817    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323839    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323845    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323849    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323854    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323858    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323863    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323868    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323872    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323877    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323881    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323885    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.328814 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323890    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323895    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323900    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323904    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323909    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323914    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323924    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323932    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323939    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323944    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323949    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323954    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323958    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323963    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323967    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323972    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323977    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323982    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323988    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.329453 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323993    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.323998    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324002    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324007    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324011    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324018    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324023    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324028    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324032    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324037    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324041    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324045    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324050    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324054    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324059    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324065    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324069    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324074    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324078    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324083    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.330026 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324088    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324093    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324098    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324102    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.324107    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325243    2564 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325267    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325286    2564 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325295    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325303    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325308    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325316    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325323    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325329    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325334    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325340    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325346    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325352    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325357    2564 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325362    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325367    2564 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325373    2564 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325378    2564 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:26:01.330613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325384    2564 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325389    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325395    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325400    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325408    2564 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325416    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325421    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325426    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325432    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325437    2564 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325443    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325449    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325454    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325459    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325467    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325472    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325477    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325483    2564 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325488    2564 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325493    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325502    2564 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325507    2564 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325513    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325517    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:26:01.331184 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325522    2564 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325530    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325536    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325541    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325546    2564 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325552    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325556    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325561    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325567    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325572    2564 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325578    2564 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325583    2564 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325592    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325598    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325603    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325608    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325614    2564 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325619    2564 flags.go:64] FLAG: --help="false"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325625    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325631    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325637    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325642    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325648    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:26:01.331868 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325654    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325660    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325664    2564 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325669    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325674    2564 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325679    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325685    2564 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325690    2564 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325695    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325700    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325705    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325710    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325715    2564 flags.go:64] FLAG: --lock-file=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325720    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325725    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325730    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325740    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325745    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325750    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325756    2564 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325761    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325768    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325773    2564 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325778    2564 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325786    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:26:01.332416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325791    2564 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325798    2564 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325804    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325809    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325814    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325819    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325839    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325844    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325849    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325861    2564 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325867    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325873    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325878    2564 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325883    2564 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325888    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325893    2564 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325898    2564 flags.go:64] FLAG: --port="10250"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325903    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325908    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f658221cc576a613"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325913    2564 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325919    2564 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325924    2564 flags.go:64] FLAG: --register-node="true"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325929    2564 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325936    2564 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325941    2564 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325945    2564 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:26:01.333081 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325951    2564 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325958    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325964    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325969    2564 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325974    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325979    2564 flags.go:64] FLAG: --runonce="false"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325984    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325989    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.325995    2564 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326000    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326005    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326011    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326019    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326024    2564 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326029    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326035    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326040    2564 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326045    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326051    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326056    2564 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326061    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326070    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326075    2564 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326080    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326087    2564 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:26:01.333776 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326092    2564 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326097    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326102    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326107    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326112    2564 flags.go:64] FLAG: --v="2"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326120    2564 flags.go:64] FLAG: --version="false"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326127    2564 flags.go:64] FLAG: --vmodule=""
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326134    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.326140    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326385    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326392    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326399    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326406    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326413    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326418    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326423    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326429    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326434    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326440    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326449    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326454    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.334394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326459    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326464    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326470    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326475    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326480    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326485    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326490    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326495    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326500    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326504    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326509    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326513    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326518    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326523    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326528    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326533    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326538    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326543    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326548    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.334917 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326553    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326558    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326562    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326567    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326572    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326578    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326583    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326587    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326592    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326596    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326602    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326608    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326614    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326619    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326626    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326633    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326638    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326643    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326649    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.335406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326654    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326659    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326664    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326669    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326674    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326678    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326683    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326687    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326692    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326696    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326701    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326727    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326735    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326740    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326746    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326750    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326755    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326760    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326765    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326769    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.335879 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326774    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326779    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326783    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326788    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326795    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326800    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326806    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326811    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326815    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326820    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326840    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326845    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326850    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326854    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326859    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326863    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326868    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326873    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326877    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326882    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.336351 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326887    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326891    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326896    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326900    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326905    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326910    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326917    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326921    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326927    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326932    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326936    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326941    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326945    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326950    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326955    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.326960    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.336877 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.327007    2564 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327077    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327084    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327092    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327096    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327101    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327106    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327112    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327117    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327121    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327126    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327131    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327136    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327141    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327146    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327151    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327155    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327160    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327166    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327171    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.337277 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327176    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327180    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327191    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327199    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327204    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327209    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327214    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327220    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327224    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327229    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327234    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327239    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327244    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327251    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327256    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327260    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327266    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327270    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327275    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.337751 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327280    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327285    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327289    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327294    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327298    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327303    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327308    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327312    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327317    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327321    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327326    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327331    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327336    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327340    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327345    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327352    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327357    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327361    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327366    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327371    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.338222 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327375    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327380    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327385    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327389    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327394    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327398    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327404    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327409    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327414    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327419    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327424    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327429    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327434    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327438    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327442    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327447    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327452    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327457    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327461    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327466    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.338680 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327471    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327475    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327480    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327485    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327490    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327494    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327502    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327510    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327515    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327519    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327523    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327528    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327532    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327536    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327541    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327546    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327550    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327555    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327561    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.339175 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327566    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327571    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327575    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327580    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327586    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327591    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327596    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327600    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.327605    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.335087    2564 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.335100    2564 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.335117    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.335122    2564 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335181    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335187    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335190    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335194    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335197    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335200    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.339623 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335203    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335207    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335209    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335212    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335215    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335217    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335220    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335223    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335225    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335228    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335230    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335234    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335236    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335239    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335242    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335244    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335247    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335250    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335253    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.340095 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335256    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335259    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335261    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335264    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335267    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335270    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335273    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335275    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335278    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335281    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335283    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335286    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335288    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335291    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335294    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335296    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335299    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335301    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335304    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.340608 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335306    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335309    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335311    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335314    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335317    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335319    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335322    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335325    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335327    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335330    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335332    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335335    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335338    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335341    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335344    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335347    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335350    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335353    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335356    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335359    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335361    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.341077 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335364    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335368    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335372    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335375    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335378    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335380    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335383    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335385    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335388    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335391    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335393    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335396    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335399    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335402    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335404    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335407    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335410    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335412    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335415    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.341558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335418    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335421    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335424    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335426    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335429    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335432    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335434    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335437    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335440    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335443    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335445    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335448    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335451    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335455    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335459    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335462    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335465    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335467    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335470    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.342096 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335473    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335475    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335478    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.335505    2564 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335657    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335662    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335665    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335668    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335671    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335674    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335676    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335679    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335681    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335686    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335691    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.342547 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335694    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335697    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335700    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335703    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335709    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335712    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335714    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335717    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335720    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335722    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335725    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335727    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335730    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335732    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335735    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335738    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335740    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335742    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335745    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335748    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.342925 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335750    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335753    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335756    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335758    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335761    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335771    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335774    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335777    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335780    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335782    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335785    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335788    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335791    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335793    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335795    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335798    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335801    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335804    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335806    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335809    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.343394 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335812    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335814    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335817    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335820    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335835    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335840    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335844    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335848    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335851    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335853    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335856    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335858    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335861    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335863    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335866    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335869    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335872    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335875    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335878    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.343922 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335881    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335884    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335887    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335889    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335892    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335894    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335897    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335900    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335903    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335905    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335907    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335911    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335914    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335918    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335921    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335924    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335927    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335929    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335932    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.344366 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335934    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335937    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335939    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335942    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335945    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335947    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335949    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335952    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335955    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335957    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335960    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335962    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335965    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335967    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335970    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335973    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.344805 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.335975    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.336000    2564 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336036    2564 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336041    2564 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336044    2564 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336046    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336049    2564 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336052    2564 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336055    2564 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336058    2564 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336060    2564 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336063    2564 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336066    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336068    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336071    2564 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336074    2564 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.345188 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336077    2564 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336079    2564 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336082    2564 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336084    2564 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336087    2564 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336089    2564 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336092    2564 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336094    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336097    2564 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336102    2564 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336105    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336107    2564 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336110    2564 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336113    2564 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336115    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336118    2564 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336120    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336123    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336126    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336128    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.345558 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336131    2564 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336133    2564 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336136    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336138    2564 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336141    2564 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336143    2564 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336146    2564 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336149    2564 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336151    2564 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336154    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336156    2564 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336159    2564 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336161    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336164    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336166    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336169    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336171    2564 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336174    2564 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336177    2564 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.346040 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336179    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336182    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336186    2564 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336188    2564 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336190    2564 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336193    2564 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336197    2564 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336201    2564 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336204    2564 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336206    2564 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336209    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336211    2564 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336214    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336216    2564 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336219    2564 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336221    2564 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336224    2564 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336226    2564 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336229    2564 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.346485 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336231    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336234    2564 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336236    2564 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336239    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336241    2564 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336244    2564 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336246    2564 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336249    2564 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336252    2564 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336254    2564 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336257    2564 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336259    2564 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336262    2564 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336265    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336267    2564 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336271    2564 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336273    2564 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336276    2564 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336278    2564 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.346944 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336282    2564 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336286    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336290    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336293    2564 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336296    2564 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336298    2564 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336302    2564 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336305    2564 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336307    2564 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336310    2564 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336312    2564 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336315    2564 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336317    2564 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336320    2564 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:01.336322    2564 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.336373    2564 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.338877    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.339995    2564 server.go:1014] "Starting client certificate rotation"
Nov 02 12:26:01.347461 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.340093    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:01.348245 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.340127    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:01.348245 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.342259    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:01.348245 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.345241    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:01.359039 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.359024    2564 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:26:01.362909 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.362895    2564 log.go:25] "Validated CRI v1 image API"
Nov 02 12:26:01.363705 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.363681    2564 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:26:01.370328 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.370307    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8568abb0-7df4-484e-96d5-9ac2e0a1c555:/dev/nvme0n1p4 b478e783-b0a2-42a4-96ee-1384324d3662:/dev/nvme0n1p3]
Nov 02 12:26:01.370397 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.370328    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 12:26:01.375546 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.375441    2564 manager.go:217] Machine: {Timestamp:2026-11-02 12:26:01.374979596 +0000 UTC m=+0.388209620 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099187 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2571b7ba6e5a25fffcd6b18ffa14cf SystemUUID:ec2571b7-ba6e-5a25-fffc-d6b18ffa14cf BootID:00570ddc-d6f1-459f-8864-9a976f362aa4 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:42:08:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:42:08:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:ea:23:0e:b6:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:26:01.375546 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.375541    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:26:01.375650 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.375617    2564 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:26:01.379360 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379338    2564 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:26:01.379491 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379363    2564 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:26:01.379532 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379493    2564 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:26:01.379532 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379501    2564 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:26:01.379532 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379517    2564 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:01.379532 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.379530    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:01.380857 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.380847    2564 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:26:01.385614 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.385602    2564 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:26:01.385735 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.385726    2564 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:26:01.386684 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.386675    2564 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:26:01.386722 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.386689    2564 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:26:01.386722 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.386699    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:26:01.386722 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.386708    2564 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:26:01.386722 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.386716    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:26:01.387071 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.387045    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:01.389682 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.389665    2564 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:26:01.396943 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.396930    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:01.397906 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.397894    2564 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:26:01.397954 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.397923    2564 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:26:01.398080 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398069    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:26:01.398116 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398085    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:26:01.398116 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398091    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:26:01.398116 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398097    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:26:01.398116 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398102    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:26:01.398116 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398108    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:26:01.398814 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398806    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:26:01.398860 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398837    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:26:01.398860 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398848    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:26:01.398860 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398855    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:26:01.398940 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398869    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:26:01.398940 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.398879    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:26:01.399755 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.399745    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:26:01.399755 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.399754    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:26:01.403724 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.403710    2564 server.go:1263] "Started kubelet"
Nov 02 12:26:01.403845 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.403787    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:26:01.403845 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.403817    2564 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:26:01.403927 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.403866    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:26:01.404415 ip-10-0-135-160 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:26:01.405381 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.405364    2564 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:26:01.408319 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.408293    2564 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:26:01.413019 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.412998    2564 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 12:26:01.415838 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.415809    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:01.416463 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.416445    2564 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:26:01.416851 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.416812    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.417284 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417269    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:26:01.417374 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417272    2564 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:26:01.417374 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417299    2564 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:26:01.417472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417399    2564 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:26:01.417472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417406    2564 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:26:01.417767 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417747    2564 factory.go:153] Registering CRI-O factory
Nov 02 12:26:01.417862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417770    2564 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:26:01.417862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417837    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:26:01.417862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417848    2564 factory.go:55] Registering systemd factory
Nov 02 12:26:01.417862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417857    2564 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:26:01.417993 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417874    2564 factory.go:103] Registering Raw factory
Nov 02 12:26:01.417993 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.417898    2564 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:26:01.419097 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.419082    2564 manager.go:319] Starting recovery of all containers
Nov 02 12:26:01.429311 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.429291    2564 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:01.430277 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.430264    2564 manager.go:324] Recovery completed
Nov 02 12:26:01.435055 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.435022    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 12:26:01.435143 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.435076    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.436049 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.434964    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e0a9efc8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.40368276 +0000 UTC m=+0.416912785,LastTimestamp:2026-11-02 12:26:01.40368276 +0000 UTC m=+0.416912785,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.436867 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.436813    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.436867 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.436860    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.436961 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.436875    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.437520 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.437454    2564 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:26:01.437520 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.437516    2564 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:26:01.437598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.437535    2564 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:26:01.437889 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.437873    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.438959 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.438943    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.439063 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.438968    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.439063 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.438978    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.439780 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.439757    2564 policy_none.go:50] "Start"
Nov 02 12:26:01.439780 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.439778    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:26:01.439905 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.439795    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:26:01.441817 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.441806    2564 policy_none.go:44] "Start"
Nov 02 12:26:01.445320 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.445288    2564 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 12:26:01.451544 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.451525    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ssjz2"
Nov 02 12:26:01.455484 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.455384    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a3ede6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436843494 +0000 UTC m=+0.450073522,LastTimestamp:2026-11-02 12:26:01.436843494 +0000 UTC m=+0.450073522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.455796 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.455698    2564 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 12:26:01.458226 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.457511    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ssjz2"
Nov 02 12:26:01.458226 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.458126    2564 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 12:26:01.471351 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.471317    2564 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:26:01.474334 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.474319    2564 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:26:01.475035 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.475021    2564 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:26:01.475073 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475036    2564 server.go:87] "Starting device plugin registration server"
Nov 02 12:26:01.475342 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475330    2564 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:26:01.475390 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475345    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:26:01.475451 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475435    2564 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:26:01.475542 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475525    2564 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:26:01.475542 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.475540    2564 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:26:01.476083 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.476064    2564 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:26:01.476156 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.476110    2564 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.489206 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.489181    2564 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 12:26:01.489278 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.489180    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a44acc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436867276 +0000 UTC m=+0.450097303,LastTimestamp:2026-11-02 12:26:01.436867276 +0000 UTC m=+0.450097303,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.505812 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.505742    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a48182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436881282 +0000 UTC m=+0.450111311,LastTimestamp:2026-11-02 12:26:01.436881282 +0000 UTC m=+0.450111311,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.540881 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.540789    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-160.ec2.internal.18e43a08e2a3ede6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a3ede6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436843494 +0000 UTC m=+0.450073522,LastTimestamp:2026-11-02 12:26:01.438955952 +0000 UTC m=+0.452185976,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.557501 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.557420    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-160.ec2.internal.18e43a08e2a44acc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a44acc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436867276 +0000 UTC m=+0.450097303,LastTimestamp:2026-11-02 12:26:01.438972056 +0000 UTC m=+0.452202081,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.576314 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.576264    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.577227 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.577206    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.577313 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.577239    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.577313 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.577255    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.577313 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.577286    2564 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.584184 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.584102    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-160.ec2.internal.18e43a08e2a48182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-160.ec2.internal.18e43a08e2a48182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-160.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.436881282 +0000 UTC m=+0.450111311,LastTimestamp:2026-11-02 12:26:01.438982809 +0000 UTC m=+0.452212834,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-160.ec2.internal,}"
Nov 02 12:26:01.603778 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.603060    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:26:01.605914 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.604402    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:26:01.605914 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.604423    2564 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:26:01.605914 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.604441    2564 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:26:01.605914 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.604476    2564 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 12:26:01.610014 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.609993    2564 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.610099 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.610016    2564 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-160.ec2.internal\": node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.617631 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.617615    2564 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:01.648048 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.648023    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.705227 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.705201    2564 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"]
Nov 02 12:26:01.705340 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.705261    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.706243 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.706226    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.706321 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.706257    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.706321 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.706272    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.708574 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.708562    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.708727 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.708713    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.708764 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.708743    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.709736 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709719    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.709810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709721    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.709810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709778    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.709810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709789    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.709810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709748    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.709975 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.709840    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.712110 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.712094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.712174 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.712125    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.712894 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.712878    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.713000 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.712907    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.713000 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.712921    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.719447 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.719430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.719521 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.719459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.719521 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.719486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.746630 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.746609    2564 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-160.ec2.internal\" not found" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.748170 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.748155    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.749136 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.749120    2564 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-160.ec2.internal\" not found" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820200 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820200 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820200    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820377 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820377 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820253    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820377 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820278    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.820377 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:01.820296    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:01.849273 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.849222    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:01.950150 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:01.950103    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.050654 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.050616    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.052786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.052766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:02.054919 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.054902    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:02.150868 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.150776    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.252390 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.252358    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.340010 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.339976    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 12:26:02.340518 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.340124    2564 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 12:26:02.353238 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.353210    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.454295 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.454205    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.502694 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.502656    2564 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:02.504458 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.504427    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:02.504604 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.504466    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:02.504604 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.504483    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:02.534846 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.534820    2564 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:02.555264 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:02.555240    2564 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:02.578471 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.578435    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:01 +0000 UTC" deadline="2028-06-19 20:52:11.059724633 +0000 UTC"
Nov 02 12:26:02.578471 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.578468    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14288h26m8.481260328s"
Nov 02 12:26:02.623173 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.623147    2564 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:02.682769 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:02.682726    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c19b4cc2c9735bf381d4249f2c0fe4.slice/crio-b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f WatchSource:0}: Error finding container b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f: Status 404 returned error can't find the container with id b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f
Nov 02 12:26:02.684591 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:02.684566    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7bed3dd43ef144d6694274c9c4b683.slice/crio-2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29 WatchSource:0}: Error finding container 2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29: Status 404 returned error can't find the container with id 2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29
Nov 02 12:26:02.691301 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.691286    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:26:02.717795 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.717724    2564 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:02.752777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.752748    2564 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:02.754708 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.754693    2564 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:02.782976 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.782956    2564 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:02.936913 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:02.936886    2564 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:03.387685 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.387654    2564 apiserver.go:52] "Watching apiserver"
Nov 02 12:26:03.419012 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.418978    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:03.421814 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.421777    2564 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:26:03.422222 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.422195    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-d8xkp","kube-system/global-pull-secret-syncer-qqgcj","openshift-dns/node-resolver-2c8j4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal","kube-system/konnectivity-agent-dwphp","openshift-network-operator/iptables-alerter-hqnhw","openshift-ovn-kubernetes/ovnkube-node-97gdw","kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph","openshift-multus/multus-22kwl","openshift-multus/multus-additional-cni-plugins-mff82","openshift-network-diagnostics/network-check-target-bx8dr","openshift-cluster-node-tuning-operator/tuned-rsbxh","openshift-image-registry/node-ca-cdn4d"]
Nov 02 12:26:03.425215 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.425194    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.427346 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.427319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.429703 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.429684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.432095 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.432068    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.436609 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.436586    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.439018 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.438996    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.441610 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.441585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.443226 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.443149    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:26:03.443226 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.443196    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:26:03.443387 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.443201    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.443387 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.443313    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:26:03.443564 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.443481    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.444172 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.444153    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.444329 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.444314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.445323 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.445283    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:26:03.445419 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.445382    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.446767 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.446749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.446877 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.446829    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:03.449294 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.449272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.449401 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.449347    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:03.452442 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.451917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.454529 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.454508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:03.454629 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.454588    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.457782    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.457807    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.457886    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458014    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458028    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458042    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458087    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458088    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458134    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458165    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458173    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458230    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458255    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:26:03.458368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458282    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:26:03.459128 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.458994    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:26:03.459128 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.459092    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:26:03.459237 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.459131    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.459237 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.459167    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.461545 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461523    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:26:03.461643 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461580    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:26:03.461871 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461851    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:26:03.461942 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461874    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:26:03.461942 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461886    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:26:03.461942 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461880    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:26:03.461942 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.461907    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:26:03.479862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.478138    2564 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:03.479862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.478207    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvpzp"
Nov 02 12:26:03.519785 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.519758    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:26:03.529071 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.529211 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.529211 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529211 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.529421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6vh\" (UniqueName: \"kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.529421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.529421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.529769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.529973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wt\" (UniqueName: \"kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.530202 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkmn\" (UniqueName: \"kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrgr\" (UniqueName: \"kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8h2\" (UniqueName: \"kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblvc\" (UniqueName: \"kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vn2h\" (UniqueName: \"kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.530820 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs5x\" (UniqueName: \"kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.530973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ppz\" (UniqueName: \"kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.531598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531717    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.532589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.533472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.531977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.533472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.532006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.533472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.532033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.533472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.532056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.533472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.532079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.551425 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.551396    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:26:03.551605 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.551448    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:26:03.551605 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.551489    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:26:03.613402 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.613332    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerStarted","Data":"b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f"}
Nov 02 12:26:03.614932 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.614888    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal" event={"ID":"fc7bed3dd43ef144d6694274c9c4b683","Type":"ContainerStarted","Data":"2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29"}
Nov 02 12:26:03.633036 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.632998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.633036 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.633241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6vh\" (UniqueName: \"kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.633659 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wt\" (UniqueName: \"kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633726    2564 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkmn\" (UniqueName: \"kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.634499 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633987    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.633994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrgr\" (UniqueName: \"kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8h2\" (UniqueName: \"kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634050    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblvc\" (UniqueName: \"kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634103    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634144    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634152    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vn2h\" (UniqueName: \"kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634255    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs5x\" (UniqueName: \"kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.635212 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ppz\" (UniqueName: \"kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634545    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.634618    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.634685    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:04.134663225 +0000 UTC m=+3.147893259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636052 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635397    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635462    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635507    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635544    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635583    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635622    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.636928 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635676    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635727    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635779    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635789    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635826    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635828    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635863    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635904    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635949    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635996    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636043    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636094    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.634724    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636224    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636307    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636480    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636521    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636621    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.637829 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636667    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636707    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.636985    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637046    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637101    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637328    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637502    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637501    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637542    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637663    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637698    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637822    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637877    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.637949    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638203    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638248    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638292    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638356    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.638777 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638418    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638448    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638471    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638526    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638555    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638569    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.635729    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638638    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638645    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.638653    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.638722    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:04.138701815 +0000 UTC m=+3.151931844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638724    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638766    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.638782    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.639034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.639098    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.639133    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.639702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.639459    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.640573 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.640322    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.640573 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.640490    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.640676 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.640614    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.641983 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.641960    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.660596 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.660566    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:03.660596 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.660595    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:03.660778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.660607    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:03.660778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:03.660662    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:04.160645638 +0000 UTC m=+3.173875699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:03.661141 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.661119    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs5x\" (UniqueName: \"kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.662865 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.662830    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vn2h\" (UniqueName: \"kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.663707 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.663658    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wt\" (UniqueName: \"kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.666123 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.665774    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6vh\" (UniqueName: \"kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.666669 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.666244    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrgr\" (UniqueName: \"kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:03.668844 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.667048    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8h2\" (UniqueName: \"kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.668844 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.667542    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkmn\" (UniqueName: \"kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.668844 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.668127    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ppz\" (UniqueName: \"kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.668844 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.668536    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblvc\" (UniqueName: \"kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.737844 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.737810    2564 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:03.743016 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.742967    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22kwl"
Nov 02 12:26:03.750932 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.750907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:03.755215 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.755191    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76295dc_1849_4d96_8940_cdc1de45551e.slice/crio-c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92 WatchSource:0}: Error finding container c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92: Status 404 returned error can't find the container with id c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92
Nov 02 12:26:03.761406 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.761378    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ee3214_5327_4181_8c06_5a9c350ed235.slice/crio-a26c47823a718c931f95fc97a8f5a9696ef4a9935f6ab3d611ada77d647b1287 WatchSource:0}: Error finding container a26c47823a718c931f95fc97a8f5a9696ef4a9935f6ab3d611ada77d647b1287: Status 404 returned error can't find the container with id a26c47823a718c931f95fc97a8f5a9696ef4a9935f6ab3d611ada77d647b1287
Nov 02 12:26:03.765887 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.765743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:03.772233 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.772059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:03.776929 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.776888    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9939d8b4_1181_463b_a465_13da054067b6.slice/crio-700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad WatchSource:0}: Error finding container 700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad: Status 404 returned error can't find the container with id 700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad
Nov 02 12:26:03.783780 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.783608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:03.784267 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.784235    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d37dd0_8676_4b17_81a4_201865d84b89.slice/crio-ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918 WatchSource:0}: Error finding container ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918: Status 404 returned error can't find the container with id ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918
Nov 02 12:26:03.793029 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.792998    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69559f09_3ce2_435a_a738_1d6892629b7d.slice/crio-f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651 WatchSource:0}: Error finding container f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651: Status 404 returned error can't find the container with id f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651
Nov 02 12:26:03.795089 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.795070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:03.803454 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.803422    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb68925_753a_4c72_8d6d_01b1e2550c6a.slice/crio-3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0 WatchSource:0}: Error finding container 3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0: Status 404 returned error can't find the container with id 3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0
Nov 02 12:26:03.809551 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.809531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:03.819344 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.819319    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e8a733_2eca_4192_a6ab_67ccc40ff2ef.slice/crio-145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141 WatchSource:0}: Error finding container 145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141: Status 404 returned error can't find the container with id 145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141
Nov 02 12:26:03.850283 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.850236    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:03.857728 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:03.857694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:03.862101 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.862065    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0b7f5d_71cf_4491_a17b_63cb47bb6441.slice/crio-3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510 WatchSource:0}: Error finding container 3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510: Status 404 returned error can't find the container with id 3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510
Nov 02 12:26:03.868605 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:03.868578    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0715df_bd6b_45ec_b51a_fb79504da0e5.slice/crio-9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82 WatchSource:0}: Error finding container 9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82: Status 404 returned error can't find the container with id 9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82
Nov 02 12:26:04.138365 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.138328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:04.138546 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.138526    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:04.138616 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.138597    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.138575745 +0000 UTC m=+4.151805778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:04.239721 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.239681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:04.239933 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.239845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:04.239933 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.239909    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:04.240062 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.239942    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:04.240062 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.239958    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:04.240062 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.240005    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:04.240062 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.240031    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.240008478 +0000 UTC m=+4.253238511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:04.240062 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.240060    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.240043782 +0000 UTC m=+4.253273799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:04.482219 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.482098    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:03 +0000 UTC" deadline="2028-07-13 09:16:55.920882122 +0000 UTC"
Nov 02 12:26:04.482219 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.482171    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14852h50m51.438717055s"
Nov 02 12:26:04.605560 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.605517    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:04.605764 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.605742    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:04.606249 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.606220    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:04.606367 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:04.606341    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:04.628545 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.628509    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22kwl" event={"ID":"a76295dc-1849-4d96-8940-cdc1de45551e","Type":"ContainerStarted","Data":"c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92"}
Nov 02 12:26:04.630690 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.630661    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510"}
Nov 02 12:26:04.632929 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.632897    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph" event={"ID":"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef","Type":"ContainerStarted","Data":"145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141"}
Nov 02 12:26:04.636795 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.636764    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwphp" event={"ID":"69559f09-3ce2-435a-a738-1d6892629b7d","Type":"ContainerStarted","Data":"f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651"}
Nov 02 12:26:04.648483 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.648393    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2c8j4" event={"ID":"9939d8b4-1181-463b-a465-13da054067b6","Type":"ContainerStarted","Data":"700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad"}
Nov 02 12:26:04.664089 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.664058    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsbxh" event={"ID":"c6ee3214-5327-4181-8c06-5a9c350ed235","Type":"ContainerStarted","Data":"a26c47823a718c931f95fc97a8f5a9696ef4a9935f6ab3d611ada77d647b1287"}
Nov 02 12:26:04.679670 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.679638    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerStarted","Data":"9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82"}
Nov 02 12:26:04.683323 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.683225    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hqnhw" event={"ID":"ceb68925-753a-4c72-8d6d-01b1e2550c6a","Type":"ContainerStarted","Data":"3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0"}
Nov 02 12:26:04.689323 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:04.689294    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdn4d" event={"ID":"a7d37dd0-8676-4b17-81a4-201865d84b89","Type":"ContainerStarted","Data":"ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918"}
Nov 02 12:26:05.147757 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.147721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:05.147985 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.147965    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:05.148057 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.148045    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:07.148025289 +0000 UTC m=+6.161255324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.248146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.248217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248343    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248359    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248371    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248415    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:07.248402444 +0000 UTC m=+6.261632471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248488    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:05.249231 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.248514    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:07.248505559 +0000 UTC m=+6.261735586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:05.483328 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.482939    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:03 +0000 UTC" deadline="2028-06-26 11:54:04.161003931 +0000 UTC"
Nov 02 12:26:05.483328 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.483234    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14447h27m58.677776863s"
Nov 02 12:26:05.609543 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.609013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:05.609543 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:05.609165    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:05.715850 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:05.715317    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal" event={"ID":"fc7bed3dd43ef144d6694274c9c4b683","Type":"ContainerStarted","Data":"e1811119e0bc63cd2c425f3e15f81b05083582da776f6eec04102f2f90f16791"}
Nov 02 12:26:06.605630 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:06.605537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:06.606152 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:06.605694    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:06.606229 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:06.606174    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:06.606335 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:06.606279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:06.740741 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:06.740703    2564 generic.go:356] "Generic (PLEG): container finished" podID="10c19b4cc2c9735bf381d4249f2c0fe4" containerID="fdfc0866d0e449fe29b2d8e8ab01f8a6b9c674815d418c3ab12233f58701fa9a" exitCode=0
Nov 02 12:26:06.741704 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:06.741678    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerDied","Data":"fdfc0866d0e449fe29b2d8e8ab01f8a6b9c674815d418c3ab12233f58701fa9a"}
Nov 02 12:26:06.786915 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:06.786839    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal" podStartSLOduration=3.78682158 podStartE2EDuration="3.78682158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:05.760887133 +0000 UTC m=+4.774117172" watchObservedRunningTime="2026-11-02 12:26:06.78682158 +0000 UTC m=+5.800051618"
Nov 02 12:26:07.162758 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.162479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:07.162758 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.162668    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:07.162758 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.162741    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:11.162721141 +0000 UTC m=+10.175951170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:07.263952 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.263908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:07.264156 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.263987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:07.264156 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264141    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:07.264281 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264213    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:11.264192833 +0000 UTC m=+10.277422851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:07.264719 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264685    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:07.264719 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264714    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:07.264875 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264730    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:07.264875 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.264783    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:11.264766696 +0000 UTC m=+10.277996715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:07.607488 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.607414    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:07.608035 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:07.607566    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:07.747455 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.747409    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerStarted","Data":"5b3c68592a85762beb455156e5ce4584d9d62a3eec9084b4a110a37e2ef43c24"}
Nov 02 12:26:07.752705 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.752674    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="ef704f8ff1a536c7f9677f1667bd0c4a367ad2963ac117900247835e6b3529cd" exitCode=0
Nov 02 12:26:07.752850 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.752718    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"ef704f8ff1a536c7f9677f1667bd0c4a367ad2963ac117900247835e6b3529cd"}
Nov 02 12:26:07.793607 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:07.793539    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" podStartSLOduration=4.793517581 podStartE2EDuration="4.793517581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:07.793248582 +0000 UTC m=+6.806478616" watchObservedRunningTime="2026-11-02 12:26:07.793517581 +0000 UTC m=+6.806747626"
Nov 02 12:26:08.605671 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.605325    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:08.605671 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:08.605439    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:08.605932 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.605771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:08.605932 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:08.605850    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:08.762453 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.762409    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwphp" event={"ID":"69559f09-3ce2-435a-a738-1d6892629b7d","Type":"ContainerStarted","Data":"4e07f1626256fe0df0bcfef11c80d80db6422dbad572408b847430c560620b8c"}
Nov 02 12:26:08.779301 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.779265    2564 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:08.779810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.779786    2564 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:08.804940 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:08.804878    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dwphp" podStartSLOduration=2.711172029 podStartE2EDuration="6.804857085s" totalImagesPullingTime="4.093685056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:08.803922016 +0000 UTC m=+7.817152053" watchObservedRunningTime="2026-11-02 12:26:08.804857085 +0000 UTC m=+7.818087122"
Nov 02 12:26:09.607927 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:09.605970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:09.607927 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:09.606203    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:09.770924 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:09.770851    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdn4d" event={"ID":"a7d37dd0-8676-4b17-81a4-201865d84b89","Type":"ContainerStarted","Data":"3c1b269c80555d59ebb03adcb3d0a6dc66a5338f2d6c80cb3fed642f5206fd31"}
Nov 02 12:26:10.605323 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:10.605261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:10.605491 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:10.605402    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:10.605884 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:10.605858    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:10.606027 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:10.605979    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:10.776460 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:10.776422    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph" event={"ID":"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef","Type":"ContainerStarted","Data":"8f0472da4c7d21c72e0c4ad5275e0c8a71a8efc98feda40b595f11f654206c19"}
Nov 02 12:26:10.776821 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:10.776533    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:11.195638 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.195599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:11.195837 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.195758    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:11.195920 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.195842    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:19.195808389 +0000 UTC m=+18.209038419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:11.296385 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.296349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:11.296545 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.296418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:11.296620 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.296575    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:11.296672 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.296642    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:19.296622719 +0000 UTC m=+18.309852753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:11.297117 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.297097    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:11.297204 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.297121    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:11.297204 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.297137    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:11.297204 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.297185    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:19.297170222 +0000 UTC m=+18.310400249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:11.590395 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.590308    2564 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:26:11.608293 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.608219    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:11.608428 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:11.608324    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:11.782250 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.782163    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22kwl" event={"ID":"a76295dc-1849-4d96-8940-cdc1de45551e","Type":"ContainerStarted","Data":"05ff5a462192faa6e67c2f69d3831cf87a97d0981131cefcf8928fadea6eb9f5"}
Nov 02 12:26:11.784810 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.784777    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph" event={"ID":"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef","Type":"ContainerStarted","Data":"4ef7261574b2662d301319b5623cac5840ca30bbbacf6f3965193bf8036137f8"}
Nov 02 12:26:11.828142 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.827508    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-22kwl" podStartSLOduration=2.116740923 podStartE2EDuration="9.827489698s" totalImagesPullingTime="7.710748775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:11.826887907 +0000 UTC m=+10.840117939" watchObservedRunningTime="2026-11-02 12:26:11.827489698 +0000 UTC m=+10.840719732"
Nov 02 12:26:11.828142 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:11.827901    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cdn4d" podStartSLOduration=4.669249392 podStartE2EDuration="9.827892027s" totalImagesPullingTime="5.158642635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:09.817165465 +0000 UTC m=+8.830395500" watchObservedRunningTime="2026-11-02 12:26:11.827892027 +0000 UTC m=+10.841122060"
Nov 02 12:26:12.480185 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.479912    2564 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:26:11.5903274Z","UUID":"8b78212b-2f3a-4149-b339-41edd0bcc038","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:26:12.481965 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.481936    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:26:12.481965 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.481964    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:26:12.605204 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.605176    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:12.605352 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.605218    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:12.605352 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:12.605328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:12.605456 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:12.605441    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:12.788965 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.788383    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2c8j4" event={"ID":"9939d8b4-1181-463b-a465-13da054067b6","Type":"ContainerStarted","Data":"ebc90c38a26c52ae93343537e034b53457a95be195b86a1772cc7dd15a099f76"}
Nov 02 12:26:12.791563 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.791533    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="d4c1ceb499f52323ec3e3c38ba0735486498a18b1c4b4c6631851b70c2db0fea" exitCode=0
Nov 02 12:26:12.791706 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.791631    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"d4c1ceb499f52323ec3e3c38ba0735486498a18b1c4b4c6631851b70c2db0fea"}
Nov 02 12:26:12.795864 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.795839    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph" event={"ID":"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef","Type":"ContainerStarted","Data":"b07005756c49b05216461729c6f8772ea5deea5eb0cd8ce30fdb7c3fb22fafb4"}
Nov 02 12:26:12.831429 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.831371    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2c8j4" podStartSLOduration=2.614185331 podStartE2EDuration="10.831358483s" totalImagesPullingTime="8.217173152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:12.830671263 +0000 UTC m=+11.843901296" watchObservedRunningTime="2026-11-02 12:26:12.831358483 +0000 UTC m=+11.844588516"
Nov 02 12:26:12.875922 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:12.875813    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph" podStartSLOduration=2.639401789 podStartE2EDuration="10.875796335s" totalImagesPullingTime="8.236394546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:12.875476588 +0000 UTC m=+11.888706621" watchObservedRunningTime="2026-11-02 12:26:12.875796335 +0000 UTC m=+11.889026373"
Nov 02 12:26:13.604959 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.604924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:13.605118 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:13.605035    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:13.799772 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.799686    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="ca6ee48be7fe69b714035bacfe99898c38e6bceb824fe1c1fd65d7216be9dd8a" exitCode=0
Nov 02 12:26:13.800293 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.799771    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"ca6ee48be7fe69b714035bacfe99898c38e6bceb824fe1c1fd65d7216be9dd8a"}
Nov 02 12:26:13.801460 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.801438    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hqnhw" event={"ID":"ceb68925-753a-4c72-8d6d-01b1e2550c6a","Type":"ContainerStarted","Data":"c7a0c0957027fb02e62ad58cabc6f9215ac9c8624f24078d3de3b34a87838c26"}
Nov 02 12:26:13.890701 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.890644    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hqnhw" podStartSLOduration=4.126427594 podStartE2EDuration="11.890628201s" totalImagesPullingTime="7.764200607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:13.890227242 +0000 UTC m=+12.903457275" watchObservedRunningTime="2026-11-02 12:26:13.890628201 +0000 UTC m=+12.903858235"
Nov 02 12:26:13.987807 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.987776    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:13.987996 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.987957    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:13.988672 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:13.988647    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:14.605164 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:14.605132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:14.605303 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:14.605165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:14.605303 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:14.605259    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:14.605397 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:14.605360    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:14.804901 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:14.804867    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsbxh" event={"ID":"c6ee3214-5327-4181-8c06-5a9c350ed235","Type":"ContainerStarted","Data":"e472b1dccac6c8f9db530b3192a3018b17920cc6bb8a00008f4ac0edefaeb1d1"}
Nov 02 12:26:14.849944 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:14.849895    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rsbxh" podStartSLOduration=2.763631208 podStartE2EDuration="12.849881247s" totalImagesPullingTime="10.086250039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:14.84980275 +0000 UTC m=+13.863032783" watchObservedRunningTime="2026-11-02 12:26:14.849881247 +0000 UTC m=+13.863111279"
Nov 02 12:26:15.604721 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:15.604688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:15.604933 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:15.604866    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:15.810109 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:15.810066    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="68f826b6c5789e33cae5932563fcc8265dee04a59f1c8d020150c6e07616d83a" exitCode=0
Nov 02 12:26:15.810779 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:15.810150    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"68f826b6c5789e33cae5932563fcc8265dee04a59f1c8d020150c6e07616d83a"}
Nov 02 12:26:16.605331 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:16.605301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:16.605528 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:16.605395    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:16.605528 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:16.605456    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:16.605528 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:16.605500    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:17.605065 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:17.604851    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:17.605551 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:17.605160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:18.605550 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:18.605521    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:18.605991 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:18.605521    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:18.605991 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:18.605642    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:18.605991 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:18.605692    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:18.820018 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:18.818042    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"fe50617007c6f48740fd9e3de567e19912c38d7589b159e8b51c215345eecfa4"}
Nov 02 12:26:18.821385 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:18.821360    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="eea4d9df0a8d78440235d95b0033f874e13f33e1cf461804c49460038d760086" exitCode=0
Nov 02 12:26:18.821496 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:18.821402    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"eea4d9df0a8d78440235d95b0033f874e13f33e1cf461804c49460038d760086"}
Nov 02 12:26:19.258588 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.258561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:19.258715 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.258673    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:19.258749 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.258717    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret podName:c5b28cf8-a271-41c8-a765-811b45bfeeb0 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.258706241 +0000 UTC m=+34.271936253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret") pod "global-pull-secret-syncer-qqgcj" (UID: "c5b28cf8-a271-41c8-a765-811b45bfeeb0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:19.358875 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.358790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:19.358875 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.358838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.358933    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.358953    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.358970    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.358976    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.358964381 +0000 UTC m=+34.372194405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.358980    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mn7nf for pod openshift-network-diagnostics/network-check-target-bx8dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:19.359076 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.359023    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf podName:45c4cc29-c98d-4fd9-b6d8-8090d72a11ab nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.359011308 +0000 UTC m=+34.372241322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mn7nf" (UniqueName: "kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf") pod "network-check-target-bx8dr" (UID: "45c4cc29-c98d-4fd9-b6d8-8090d72a11ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:19.605441 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.605411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:19.605611 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:19.605515    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:19.826972 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.826934    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"befb99f504a0190ebb30b5c541d14be9cc9b74803837dde242228b86e09389df"}
Nov 02 12:26:19.827145 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.827021    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"8ec3e354a79e1254b6893a6f6080663b59201b735d37ba04383040ed57beaa23"}
Nov 02 12:26:19.827145 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.827065    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"cb3761abc0299a2d2f89ffeb483fc86170f47f134ce92522cbe9440f4692edb0"}
Nov 02 12:26:19.827145 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.827107    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"8890a23884676ef8746844da643b8bf6da96617408a25a9c97efaf27782b658d"}
Nov 02 12:26:19.827276 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.827150    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"8b2e65b94a1bd7b402e8a601e824234d17e0c2b0704929777f5937f5c640423c"}
Nov 02 12:26:19.829795 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.829766    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="4d3f271442145d98b63d017877545c45dc1967f4a7044c49ed7cc37000e6a3f9" exitCode=0
Nov 02 12:26:19.829911 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:19.829801    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerDied","Data":"4d3f271442145d98b63d017877545c45dc1967f4a7044c49ed7cc37000e6a3f9"}
Nov 02 12:26:20.605131 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:20.605098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:20.605287 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:20.605105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:20.605287 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:20.605200    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:20.605287 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:20.605254    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:20.834888 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:20.834850    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mff82" event={"ID":"2d0715df-bd6b-45ec-b51a-fb79504da0e5","Type":"ContainerStarted","Data":"03e666117180ef02fd55e8c376ca264811f35edc35e4b2ff56ec33f1c58de2f9"}
Nov 02 12:26:20.911841 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:20.911736    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mff82" podStartSLOduration=7.997602471 podStartE2EDuration="18.91172188s" totalImagesPullingTime="10.914119409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:20.911299783 +0000 UTC m=+19.924529816" watchObservedRunningTime="2026-11-02 12:26:20.91172188 +0000 UTC m=+19.924951945"
Nov 02 12:26:21.607562 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:21.607489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:21.607698 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:21.607583    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:21.840493 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:21.840458    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"9c6f6f6828db2cbf138f66acb5b5e071be87262b23a3b9eb1118867791157c3c"}
Nov 02 12:26:22.605419 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:22.605387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:22.605606 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:22.605428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:22.605606 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:22.605487    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:22.605687 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:22.605606    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:23.605747 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:23.605678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:23.606288 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:23.605792    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:23.848261 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:23.848079    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" event={"ID":"ee0b7f5d-71cf-4491-a17b-63cb47bb6441","Type":"ContainerStarted","Data":"eaab173ffbc030014bf5c25b2f5247607ff7e0011dc388eaae6e5d03406dc734"}
Nov 02 12:26:23.848415 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:23.848311    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:23.863903 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:23.863794    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:23.942011 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:23.941959    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw" podStartSLOduration=7.077191751 podStartE2EDuration="21.941944794s" totalImagesPullingTime="14.864753043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:23.928739105 +0000 UTC m=+22.941969140" watchObservedRunningTime="2026-11-02 12:26:23.941944794 +0000 UTC m=+22.955174848"
Nov 02 12:26:24.605156 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:24.605127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:24.605292 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:24.605229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:24.605328 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:24.605284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:24.605401 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:24.605382    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:24.849816 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:24.849785    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:24.850241 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:24.850223    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:24.865163 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:24.865100    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:25.372136 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.372101    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bx8dr"]
Nov 02 12:26:25.372374 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.372222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:25.372374 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:25.372333    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:25.375334 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.375140    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqgcj"]
Nov 02 12:26:25.375334 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.375223    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:25.375334 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:25.375304    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:25.375800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.375779    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8xkp"]
Nov 02 12:26:25.375902 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.375891    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:25.375980 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:25.375965    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:25.851983 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:25.851906    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:26.854025 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:26.853998    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:27.607956 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:27.607877    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:27.607956 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:27.607948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:27.608168 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:27.607966    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:27.608168 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:27.608077    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:27.608259 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:27.608179    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:27.608259 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:27.608236    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:29.609680 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:29.609649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:29.610365 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:29.609654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:29.610365 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:29.609747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bx8dr" podUID="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab"
Nov 02 12:26:29.610365 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:29.609845    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8xkp" podUID="66807b2c-d92d-48aa-815d-4b437efae8eb"
Nov 02 12:26:29.610365 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:29.609655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:29.610365 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:29.609918    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqgcj" podUID="c5b28cf8-a271-41c8-a765-811b45bfeeb0"
Nov 02 12:26:31.245264 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.245239    2564 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeReady"
Nov 02 12:26:31.245673 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.245361    2564 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 12:26:31.299168 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.299137    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzn62"]
Nov 02 12:26:31.306756 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.306730    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:26:31.306927 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.306906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:31.311446 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.311425    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m5bqj"]
Nov 02 12:26:31.311558 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.311538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.316059 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.316006    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9nkss"]
Nov 02 12:26:31.317587 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.317562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:31.324444 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.324423    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:31.324559 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.324479    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86d86d55f7-vpzjg"]
Nov 02 12:26:31.324559 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.324539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:31.324666 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.324635    2564 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:31.327682 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.327665    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"]
Nov 02 12:26:31.327801 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.327784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.329706 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.329686    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"]
Nov 02 12:26:31.329925 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.329908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:31.331797 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.331777    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.331959 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.331928    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.333133 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.333112    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8gt88"]
Nov 02 12:26:31.333543 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.333525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:31.335727 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.335710    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:31.335938 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.335917    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-55ghk"]
Nov 02 12:26:31.336349 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.335788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.338472 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.338449    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"]
Nov 02 12:26:31.339974 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.338883    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 12:26:31.339974 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.338943    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 12:26:31.339974 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.338991    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnbq6\""
Nov 02 12:26:31.339974 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.339121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.341979 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.341959    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:26:31.342076 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.342061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.349118 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.349099    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"]
Nov 02 12:26:31.350218 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.350198    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.353712 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.353681    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"]
Nov 02 12:26:31.353977 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.353957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.359193 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.359174    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-x6zzd"]
Nov 02 12:26:31.360657 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.359446    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.364129 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.364111    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"]
Nov 02 12:26:31.371146 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.368935    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.374862 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.374843    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2928m"]
Nov 02 12:26:31.374959 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.374937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.377490 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.377469    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qsffd"]
Nov 02 12:26:31.377598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.377584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.380162 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380143    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzn62"]
Nov 02 12:26:31.380255 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380171    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m5bqj"]
Nov 02 12:26:31.380255 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380192    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9nkss"]
Nov 02 12:26:31.380255 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380214    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:26:31.380255 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380236    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"]
Nov 02 12:26:31.380255 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380257    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-55ghk"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380280    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380301    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380330    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-x6zzd"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380351    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380375    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qsffd"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380398    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"]
Nov 02 12:26:31.380424 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380419    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86d86d55f7-vpzjg"]
Nov 02 12:26:31.380738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380446    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"]
Nov 02 12:26:31.380738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.380467    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8gt88"]
Nov 02 12:26:31.385598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.385575    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:26:31.386343 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.386323    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2928m"]
Nov 02 12:26:31.430142 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430110    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.430142 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430136    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.430443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430419    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.430502 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430455    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 12:26:31.430559 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430527    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 12:26:31.430615 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430582    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 12:26:31.430710 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430693    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.430779 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430770    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.430896 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.430882    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.431150 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431131    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.431269 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431249    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.431341 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431274    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.431341 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431284    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.431341 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431236    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.431341 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431297    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.431341 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431223    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:26:31.431589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431329    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 12:26:31.431589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431210    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.431589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431375    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.431589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431381    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:26:31.431589 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.431175    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.443150 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.443219 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.443260 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.443293 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68d7\" (UniqueName: \"kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.443293 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.443375 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.443416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.443416 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.443511 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.443511 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnl7\" (UniqueName: \"kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.443511 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.443619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbqt\" (UniqueName: \"kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt\") pod \"kas-connection-checker-685895c7c9-9nkss\" (UID: \"0e8b0b9d-80b8-460e-98d3-189c240cd20d\") " pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:31.443619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.443715 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76chf\" (UniqueName: \"kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf\") pod \"kas-connection-checker-685895c7c9-hzn62\" (UID: \"ce9a1148-d94a-47ad-8b2d-147af29a5797\") " pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:31.443715 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qt9\" (UniqueName: \"kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9\") pod \"network-check-source-6f78d59f99-w9hwg\" (UID: \"f2849f75-3d38-49d2-8372-fd7776ddf12e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:31.443811 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.443811 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.443945 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.443945 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.443945 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6vg\" (UniqueName: \"kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.443945 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.444178 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.443962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.444178 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.444264 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444170    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.444264 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.444361 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444316    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.444419 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.444527 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.444624 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqrj\" (UniqueName: \"kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.444624 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444606    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444647    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444668    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444669    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444721    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.444786 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.445136 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444805    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 12:26:31.445312 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnr47\" (UniqueName: \"kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.445374 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.445374 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.445470 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.445470 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.444796    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 12:26:31.445470 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445288    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 12:26:31.445619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.445619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445524    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:26:31.445619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445523    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 12:26:31.445619 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445537    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 12:26:31.445800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445627    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:26:31.445800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.445800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.445800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445717    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.445800 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.446043 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.446043 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.446043 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.445979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvpt\" (UniqueName: \"kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt\") pod \"volume-data-source-validator-58c9f5f46d-xwbjh\" (UID: \"8250eede-bd12-46d3-a569-6a242bbfd566\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:31.446043 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrnh\" (UniqueName: \"kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.446189 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.446252 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.446304 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkwq\" (UniqueName: \"kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq\") pod \"kas-connection-checker-685895c7c9-m5bqj\" (UID: \"04856dfa-3c20-45b0-bc5f-f68896f87c51\") " pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:31.446343 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.446435 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.446435 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446420    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 12:26:31.446506 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.446545 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.446597 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.446663 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.446743 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9xd\" (UniqueName: \"kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.446801 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.446772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7v44\" (UniqueName: \"kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.449285 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449182    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 12:26:31.449285 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449260    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 12:26:31.449401 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449297    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 12:26:31.449401 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449323    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 12:26:31.449401 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449339    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 12:26:31.449401 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449372    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 12:26:31.449401 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449398    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 12:26:31.449634 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.449414    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 12:26:31.457387 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.457369    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:26:31.457387 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.457385    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gp6gb\""
Nov 02 12:26:31.458171 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.458155    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:26:31.458171 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.458154    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wx6z7\""
Nov 02 12:26:31.458305 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.458198    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-htk9k\""
Nov 02 12:26:31.458369 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.458345    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 12:26:31.461221 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.461192    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 12:26:31.461403 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.461379    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:26:31.547598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.547598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.547598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.547598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqrj\" (UniqueName: \"kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.547712    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr47\" (UniqueName: \"kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.547785    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.047761979 +0000 UTC m=+31.060992002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547939    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.547948 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.547976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548031    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvpt\" (UniqueName: \"kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt\") pod \"volume-data-source-validator-58c9f5f46d-xwbjh\" (UID: \"8250eede-bd12-46d3-a569-6a242bbfd566\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrnh\" (UniqueName: \"kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkwq\" (UniqueName: \"kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq\") pod \"kas-connection-checker-685895c7c9-m5bqj\" (UID: \"04856dfa-3c20-45b0-bc5f-f68896f87c51\") " pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.548738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9xd\" (UniqueName: \"kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7v44\" (UniqueName: \"kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68d7\" (UniqueName: \"kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnl7\" (UniqueName: \"kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.549443 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbqt\" (UniqueName: \"kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt\") pod \"kas-connection-checker-685895c7c9-9nkss\" (UID: \"0e8b0b9d-80b8-460e-98d3-189c240cd20d\") " pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76chf\" (UniqueName: \"kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf\") pod \"kas-connection-checker-685895c7c9-hzn62\" (UID: \"ce9a1148-d94a-47ad-8b2d-147af29a5797\") " pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qt9\" (UniqueName: \"kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9\") pod \"network-check-source-6f78d59f99-w9hwg\" (UID: \"f2849f75-3d38-49d2-8372-fd7776ddf12e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548896    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548897    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.548919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.549957    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6vg\" (UniqueName: \"kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.550239 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.550870 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550649    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.550965 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.550944    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.551368 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.551345    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552173    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552233    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552322    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552332    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552374    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552441    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552456    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552462    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552514    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.052496698 +0000 UTC m=+31.065726722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552524    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552570    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.052555574 +0000 UTC m=+31.065785594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.052719492 +0000 UTC m=+31.065949517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552794    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:31.553476 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.552850    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.052818944 +0000 UTC m=+31.066048969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552961    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553149    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.552966    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553226    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553242    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553289    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.053275534 +0000 UTC m=+31.066505567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553301    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553423    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553589    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.553716    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553807    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553839    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553888    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.053870965 +0000 UTC m=+31.067100999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553913    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.553928    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:31.554223 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:31.554013    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.053996997 +0000 UTC m=+31.067227023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:31.554778 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.554429    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.554778 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.554674    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.554895 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.554854    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.555506 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.555483    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.556148 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.556130    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.556583 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.556564    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.557493 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.557474    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.574611 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.574592    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.574611 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.574603    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqrj\" (UniqueName: \"kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.577624 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.577607    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr47\" (UniqueName: \"kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.580571 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.580544    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qt9\" (UniqueName: \"kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9\") pod \"network-check-source-6f78d59f99-w9hwg\" (UID: \"f2849f75-3d38-49d2-8372-fd7776ddf12e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:31.581092 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.581061    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:31.581419 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.581398    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkwq\" (UniqueName: \"kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq\") pod \"kas-connection-checker-685895c7c9-m5bqj\" (UID: \"04856dfa-3c20-45b0-bc5f-f68896f87c51\") " pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:31.582307 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.582285    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.582561 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.582497    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvpt\" (UniqueName: \"kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt\") pod \"volume-data-source-validator-58c9f5f46d-xwbjh\" (UID: \"8250eede-bd12-46d3-a569-6a242bbfd566\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:31.582561 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.582497    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7v44\" (UniqueName: \"kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:31.583324 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.583296    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76chf\" (UniqueName: \"kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf\") pod \"kas-connection-checker-685895c7c9-hzn62\" (UID: \"ce9a1148-d94a-47ad-8b2d-147af29a5797\") " pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:31.584110 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.584087    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:31.584533 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.584513    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnl7\" (UniqueName: \"kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:31.584750 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.584730    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrnh\" (UniqueName: \"kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.584875 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.584855    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbqt\" (UniqueName: \"kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt\") pod \"kas-connection-checker-685895c7c9-9nkss\" (UID: \"0e8b0b9d-80b8-460e-98d3-189c240cd20d\") " pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:31.585608 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.585561    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68d7\" (UniqueName: \"kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:31.586090 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.586067    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6vg\" (UniqueName: \"kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:31.586237 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.586220    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9xd\" (UniqueName: \"kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.587483 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.587463    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:31.605530 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.605505    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:31.605613 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.605527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:31.605710 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.605695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:31.624011 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.623994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:31.627050 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.627026    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:26:31.627134 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.627104    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:26:31.627170 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.627162    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:26:31.627208 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.627170    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:26:31.627208 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.627186    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:26:31.643277 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.643253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:31.653373 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.653357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:31.679719 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.679696    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:31.689082 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.689060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:31.703042 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.703002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:31.709207 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.709180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:31.723013 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.722986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:31.737117 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.737091    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:31.755146 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:31.755122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:32.034980 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.034946    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzn62"]
Nov 02 12:26:32.040521 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.038200    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m5bqj"]
Nov 02 12:26:32.040521 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.039233    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a1148_d94a_47ad_8b2d_147af29a5797.slice/crio-19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752 WatchSource:0}: Error finding container 19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752: Status 404 returned error can't find the container with id 19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752
Nov 02 12:26:32.040521 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.039366    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9nkss"]
Nov 02 12:26:32.040521 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.040285    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"]
Nov 02 12:26:32.041236 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.041200    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04856dfa_3c20_45b0_bc5f_f68896f87c51.slice/crio-5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3 WatchSource:0}: Error finding container 5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3: Status 404 returned error can't find the container with id 5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3
Nov 02 12:26:32.043373 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.043351    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"]
Nov 02 12:26:32.046441 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.046004    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8250eede_bd12_46d3_a569_6a242bbfd566.slice/crio-2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4 WatchSource:0}: Error finding container 2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4: Status 404 returned error can't find the container with id 2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4
Nov 02 12:26:32.054768 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.054737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:32.054891 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.054797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:32.054891 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.054867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054899    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.054913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054914    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054970    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054978    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.05496097 +0000 UTC m=+32.068190995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.055014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:32.055021 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055024    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055006966 +0000 UTC m=+32.068236997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054970    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055058    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.055073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055093    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055081703 +0000 UTC m=+32.068311732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055092    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.055121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055135    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055123564 +0000 UTC m=+32.068353589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055144    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055167    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.054918    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055182    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055169806 +0000 UTC m=+32.068399818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.055203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055213    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055202674 +0000 UTC m=+32.068432707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055265    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055256632 +0000 UTC m=+32.068486643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:32.055378 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:32.055282    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.055274342 +0000 UTC m=+32.068504353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:32.109069 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.109033    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-x6zzd"]
Nov 02 12:26:32.115006 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.113879    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-55ghk"]
Nov 02 12:26:32.117977 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.117923    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da4be61_ac40_4531_9357_e05f84219250.slice/crio-a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764 WatchSource:0}: Error finding container a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764: Status 404 returned error can't find the container with id a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764
Nov 02 12:26:32.155449 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.155423    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"]
Nov 02 12:26:32.158860 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.158648    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qsffd"]
Nov 02 12:26:32.158860 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.158762    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ae75b6_203e_4d60_b728_412645e0a4a8.slice/crio-0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4 WatchSource:0}: Error finding container 0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4: Status 404 returned error can't find the container with id 0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4
Nov 02 12:26:32.162315 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.162289    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086665ff_9fcc_48af_982e_59bb0239605b.slice/crio-d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc WatchSource:0}: Error finding container d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc: Status 404 returned error can't find the container with id d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc
Nov 02 12:26:32.188796 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.188774    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"]
Nov 02 12:26:32.190898 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:32.190872    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5dcaa3_4ad1_4586_ad24_b8d7bb74e151.slice/crio-b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db WatchSource:0}: Error finding container b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db: Status 404 returned error can't find the container with id b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db
Nov 02 12:26:32.875944 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.875861    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qsffd" event={"ID":"086665ff-9fcc-48af-982e-59bb0239605b","Type":"ContainerStarted","Data":"d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc"}
Nov 02 12:26:32.877690 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.877661    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl" event={"ID":"10ae75b6-203e-4d60-b728-412645e0a4a8","Type":"ContainerStarted","Data":"0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4"}
Nov 02 12:26:32.879607 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.879576    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hzn62" event={"ID":"ce9a1148-d94a-47ad-8b2d-147af29a5797","Type":"ContainerStarted","Data":"9b11e9c883ad5a39826d4f759ec4b99357e0fc0ca78a7ea4245044693d418b05"}
Nov 02 12:26:32.879702 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.879624    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hzn62" event={"ID":"ce9a1148-d94a-47ad-8b2d-147af29a5797","Type":"ContainerStarted","Data":"19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752"}
Nov 02 12:26:32.882853 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.882810    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4" event={"ID":"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151","Type":"ContainerStarted","Data":"b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db"}
Nov 02 12:26:32.884607 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.884581    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-x6zzd" event={"ID":"6da4be61-ac40-4531-9357-e05f84219250","Type":"ContainerStarted","Data":"a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764"}
Nov 02 12:26:32.886586 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.886522    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerStarted","Data":"561894cb2ed866884ca46542463138c4433f924191afb90055696f3dffd53f2a"}
Nov 02 12:26:32.890848 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.890468    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m5bqj" event={"ID":"04856dfa-3c20-45b0-bc5f-f68896f87c51","Type":"ContainerStarted","Data":"14a79a4e9e116baeb7b14df72d2e2781f9a15d7d6b9c86db72eec3e5b88fa2da"}
Nov 02 12:26:32.890848 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.890511    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m5bqj" event={"ID":"04856dfa-3c20-45b0-bc5f-f68896f87c51","Type":"ContainerStarted","Data":"5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3"}
Nov 02 12:26:32.892552 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.892498    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg" event={"ID":"f2849f75-3d38-49d2-8372-fd7776ddf12e","Type":"ContainerStarted","Data":"be565a3159839d56a4ddfc70b5cd2fa0082d5d653a239069da1ead0cf57d4457"}
Nov 02 12:26:32.894691 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.894667    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh" event={"ID":"8250eede-bd12-46d3-a569-6a242bbfd566","Type":"ContainerStarted","Data":"2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4"}
Nov 02 12:26:32.898547 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.898522    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9nkss" event={"ID":"0e8b0b9d-80b8-460e-98d3-189c240cd20d","Type":"ContainerStarted","Data":"e26245873a43ad6607cc6037c0ec7ce92ce3b8d4ece822f1a90c7f8a5b776f3f"}
Nov 02 12:26:32.898643 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.898568    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9nkss" event={"ID":"0e8b0b9d-80b8-460e-98d3-189c240cd20d","Type":"ContainerStarted","Data":"a7c2ac9013838830d1ada98474e0ec0a5f8ece0aee142fab629a4ff6b617816b"}
Nov 02 12:26:32.925201 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:32.924564    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hzn62" podStartSLOduration=409.924545116 podStartE2EDuration="6m49.924545116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:32.923142436 +0000 UTC m=+31.936372470" watchObservedRunningTime="2026-11-02 12:26:32.924545116 +0000 UTC m=+31.937775151"
Nov 02 12:26:33.021420 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.021352    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-m5bqj" podStartSLOduration=410.021332758 podStartE2EDuration="6m50.021332758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:33.021217378 +0000 UTC m=+32.034447410" watchObservedRunningTime="2026-11-02 12:26:33.021332758 +0000 UTC m=+32.034562790"
Nov 02 12:26:33.022183 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.022142    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9nkss" podStartSLOduration=410.022128555 podStartE2EDuration="6m50.022128555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:32.969470746 +0000 UTC m=+31.982700771" watchObservedRunningTime="2026-11-02 12:26:33.022128555 +0000 UTC m=+32.035358590"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:33.064869    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.064995    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.064982984 +0000 UTC m=+34.078213008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065209    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065224    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065255    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065266    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.065251965 +0000 UTC m=+34.078481994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065264    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065295    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.065283944 +0000 UTC m=+34.078513967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065319    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:33.066899 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065329    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065342    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.065322894 +0000 UTC m=+34.078552908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065356    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065364    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.06535476 +0000 UTC m=+34.078584772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065390    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.06538101 +0000 UTC m=+34.078611033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065397    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065427    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.065418028 +0000 UTC m=+34.078648061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065433    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:33.067778 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:33.065478    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.065466533 +0000 UTC m=+34.078696557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:34.909421 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:34.909381    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh" event={"ID":"8250eede-bd12-46d3-a569-6a242bbfd566","Type":"ContainerStarted","Data":"3c5d99fb8b71148ea995142b5535001753e1a8ec30f7adb16197758b88090398"}
Nov 02 12:26:34.953462 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:34.953114    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh" podStartSLOduration=359.584670533 podStartE2EDuration="6m1.953100443s" totalImagesPullingTime="2.36842991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:34.951730893 +0000 UTC m=+33.964960919" watchObservedRunningTime="2026-11-02 12:26:34.953100443 +0000 UTC m=+33.966330477"
Nov 02 12:26:35.087863 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.087755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:35.087863 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.087837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.087893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.087932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.087959    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088026    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088010239 +0000 UTC m=+38.101240266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.087959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:35.088107 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088026    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088116    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088105337 +0000 UTC m=+38.101335364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088121    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.088147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088189    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088169696 +0000 UTC m=+38.101399727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088055    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088218    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088223    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088213522 +0000 UTC m=+38.101443534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.088239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088263    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088248884 +0000 UTC m=+38.101478912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088056    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.088309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088304    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088357    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088363    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088345753 +0000 UTC m=+38.101575766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088391    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088379384 +0000 UTC m=+38.101609428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:35.088400 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088402    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:35.089092 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.088439    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.088427449 +0000 UTC m=+38.101657465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:35.290847 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.290773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:35.293201 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.293166    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:35.391598 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.391559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:35.391769 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.391715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:35.391946 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.391924    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 12:26:35.392048 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:35.391994    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:27:07.391976399 +0000 UTC m=+66.405206414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : secret "metrics-daemon-secret" not found
Nov 02 12:26:35.396183 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.396157    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:35.557653 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.557574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:35.568676 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.568485    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:35.891551 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.891458    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqgcj"]
Nov 02 12:26:35.900332 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:35.900301    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b28cf8_a271_41c8_a765_811b45bfeeb0.slice/crio-e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca WatchSource:0}: Error finding container e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca: Status 404 returned error can't find the container with id e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca
Nov 02 12:26:35.903880 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.903739    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bx8dr"]
Nov 02 12:26:35.907268 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:35.907239    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c4cc29_c98d_4fd9_b6d8_8090d72a11ab.slice/crio-e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7 WatchSource:0}: Error finding container e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7: Status 404 returned error can't find the container with id e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7
Nov 02 12:26:35.914863 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.914813    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qqgcj" event={"ID":"c5b28cf8-a271-41c8-a765-811b45bfeeb0","Type":"ContainerStarted","Data":"e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca"}
Nov 02 12:26:35.919649 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:35.919586    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bx8dr" event={"ID":"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab","Type":"ContainerStarted","Data":"e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7"}
Nov 02 12:26:36.924977 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.924242    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-x6zzd" event={"ID":"6da4be61-ac40-4531-9357-e05f84219250","Type":"ContainerStarted","Data":"045f12b705abbcb2709ea1b6807dd0e9facac15e4475bc08e687eedc42124cd9"}
Nov 02 12:26:36.927747 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.927727    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/0.log"
Nov 02 12:26:36.927840 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.927767    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e2649ce-6913-40b6-a47a-cd71b5208dd3" containerID="5e12776ae1f5f963b5903f6b0e3ff73e9f813bd675cac6b28e20667aab1bde3e" exitCode=255
Nov 02 12:26:36.927892 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.927849    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerDied","Data":"5e12776ae1f5f963b5903f6b0e3ff73e9f813bd675cac6b28e20667aab1bde3e"}
Nov 02 12:26:36.928175 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.928146    2564 scope.go:122] "RemoveContainer" containerID="5e12776ae1f5f963b5903f6b0e3ff73e9f813bd675cac6b28e20667aab1bde3e"
Nov 02 12:26:36.969784 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:36.969724    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-x6zzd" podStartSLOduration=357.71683297 podStartE2EDuration="6m1.969690038s" totalImagesPullingTime="4.252857068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:36.968528583 +0000 UTC m=+35.981758616" watchObservedRunningTime="2026-11-02 12:26:36.969690038 +0000 UTC m=+35.982920072"
Nov 02 12:26:37.448587 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.447250    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qnhns"]
Nov 02 12:26:37.467802 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.467767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.473561 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.473495    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnhns"]
Nov 02 12:26:37.487709 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.487682    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:26:37.488375 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.488333    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:26:37.488480 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.488460    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:26:37.615982 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.615034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkfd\" (UniqueName: \"kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.615982 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.615090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.615982 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.615156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.615982 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.615260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.615982 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.615364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716144 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.716104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkfd\" (UniqueName: \"kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716312 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.716158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716312 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.716187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716312 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.716299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716405 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.716389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.716560 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:37.716546    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:37.716623 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:37.716613    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:38.216594479 +0000 UTC m=+37.229824506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:37.717090 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:37.717073    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:37.717133 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:37.717124    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:38.217109766 +0000 UTC m=+37.230339792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:37.717435 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.717418    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.717503 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.717490    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.741738 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.741706    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkfd\" (UniqueName: \"kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:37.936190 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.935698    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl" event={"ID":"10ae75b6-203e-4d60-b728-412645e0a4a8","Type":"ContainerStarted","Data":"84d44cda47bc7b26379c51e7aa2ba9b51c34b36a0aabeaaffd5e518fe5e35938"}
Nov 02 12:26:37.938105 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.937560    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4" event={"ID":"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151","Type":"ContainerStarted","Data":"90ebfa19ebc3d970b0e6e33bb927f3d39252b2d420e2972b1831c32fef7e4214"}
Nov 02 12:26:37.940439 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.940411    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:37.940878 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.940855    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/0.log"
Nov 02 12:26:37.940983 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.940892    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e2649ce-6913-40b6-a47a-cd71b5208dd3" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b" exitCode=255
Nov 02 12:26:37.942164 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.941585    2564 scope.go:122] "RemoveContainer" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"
Nov 02 12:26:37.942164 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:37.941751    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:37.942164 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.941993    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerDied","Data":"b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"}
Nov 02 12:26:37.942164 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.942033    2564 scope.go:122] "RemoveContainer" containerID="5e12776ae1f5f963b5903f6b0e3ff73e9f813bd675cac6b28e20667aab1bde3e"
Nov 02 12:26:37.990894 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:37.990838    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl" podStartSLOduration=357.895771175 podStartE2EDuration="6m2.990803943s" totalImagesPullingTime="5.095032768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:37.989682015 +0000 UTC m=+37.002912050" watchObservedRunningTime="2026-11-02 12:26:37.990803943 +0000 UTC m=+37.004033977"
Nov 02 12:26:38.089559 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.089495    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4" podStartSLOduration=358.370082467 podStartE2EDuration="6m3.089472816s" totalImagesPullingTime="4.719390349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:38.034057965 +0000 UTC m=+37.047287999" watchObservedRunningTime="2026-11-02 12:26:38.089472816 +0000 UTC m=+37.102702851"
Nov 02 12:26:38.234027 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.233986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:38.234213 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.234080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:38.234268 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:38.234231    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:38.234313 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:38.234281    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.234269123 +0000 UTC m=+38.247499146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:38.234629 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:38.234612    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:38.234682 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:38.234658    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.234648029 +0000 UTC m=+38.247878052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:38.421059 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.420981    2564 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"]
Nov 02 12:26:38.440326 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.440244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:38.446758 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.446723    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"]
Nov 02 12:26:38.456974 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.456941    2564 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r2cgm\""
Nov 02 12:26:38.505017 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.504931    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:38.505195 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.505175    2564 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 12:26:38.537673 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.537640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvx4\" (UniqueName: \"kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4\") pod \"migrator-66df5d496b-vq98f\" (UID: \"e51414af-06a2-416d-9656-871c483ba22a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:38.638854 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.638787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvx4\" (UniqueName: \"kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4\") pod \"migrator-66df5d496b-vq98f\" (UID: \"e51414af-06a2-416d-9656-871c483ba22a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:38.665519 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.665482    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvx4\" (UniqueName: \"kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4\") pod \"migrator-66df5d496b-vq98f\" (UID: \"e51414af-06a2-416d-9656-871c483ba22a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:38.766146 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.766107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:38.947055 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.947023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:38.947492 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.947378    2564 scope.go:122] "RemoveContainer" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"
Nov 02 12:26:38.947560 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:38.947537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:38.949370 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.949339    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg" event={"ID":"f2849f75-3d38-49d2-8372-fd7776ddf12e","Type":"ContainerStarted","Data":"95b3b2b141b1f835350ead8e0a4fce269985cd0348d10dededb60aff7da6867e"}
Nov 02 12:26:38.952352 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:38.951876    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qsffd" event={"ID":"086665ff-9fcc-48af-982e-59bb0239605b","Type":"ContainerStarted","Data":"40c7853ed3c8d8f127f2d320d23380fb3502abc8c239e231105eadd4054fa5fe"}
Nov 02 12:26:39.037402 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.037302    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qsffd" podStartSLOduration=2.242114564 podStartE2EDuration="8.037271007s" totalImagesPullingTime="5.795156443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:39.037139218 +0000 UTC m=+38.050369251" watchObservedRunningTime="2026-11-02 12:26:39.037271007 +0000 UTC m=+38.050501040"
Nov 02 12:26:39.083629 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.083596    2564 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"]
Nov 02 12:26:39.085404 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.085154    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg" podStartSLOduration=357.175582533 podStartE2EDuration="6m3.085135914s" totalImagesPullingTime="5.909553381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:39.084907617 +0000 UTC m=+38.098137647" watchObservedRunningTime="2026-11-02 12:26:39.085135914 +0000 UTC m=+38.098365948"
Nov 02 12:26:39.089356 ip-10-0-135-160 kubenswrapper[2564]: W1102 12:26:39.089325    2564 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51414af_06a2_416d_9656_871c483ba22a.slice/crio-00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75 WatchSource:0}: Error finding container 00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75: Status 404 returned error can't find the container with id 00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75
Nov 02 12:26:39.147801 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:39.147984 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:39.147984 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:39.147984 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:39.147984 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:39.147984 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:39.148398 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.147987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:39.148398 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.148033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:39.148398 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148219    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:39.148398 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148282    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.148265258 +0000 UTC m=+46.161495282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:39.148670 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148652    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:39.148740 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148708    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.148693444 +0000 UTC m=+46.161923470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:39.148803 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148770    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148883    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148927    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.148914228 +0000 UTC m=+46.162144254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148951    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148971    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148986    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149014    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.14900498 +0000 UTC m=+46.162235005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.148948    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149027    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.149019549 +0000 UTC m=+46.162249567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149057    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.149040658 +0000 UTC m=+46.162270686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149072    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.149066514 +0000 UTC m=+46.162296526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149057    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:39.149134 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.149101    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.149092706 +0000 UTC m=+46.162322716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:39.248759 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.248720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:39.248937 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.248789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:39.248937 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.248852    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:39.248937 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.248917    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:39.249071 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.248928    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:41.248908995 +0000 UTC m=+40.262139018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:39.249071 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:39.248987    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:41.248969209 +0000 UTC m=+40.262199220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:39.957438 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.957401    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bx8dr" event={"ID":"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab","Type":"ContainerStarted","Data":"d55f7b5205960529a5984c008414717970ccc842a21dbdd2c1165a6d1d7fb900"}
Nov 02 12:26:39.957961 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.957537    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:39.959441 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:39.959036    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f" event={"ID":"e51414af-06a2-416d-9656-871c483ba22a","Type":"ContainerStarted","Data":"00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75"}
Nov 02 12:26:40.002054 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:40.000081    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bx8dr" podStartSLOduration=34.854732181 podStartE2EDuration="38.000059611s" totalImagesPullingTime="3.14532743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:39.999751921 +0000 UTC m=+39.012981955" watchObservedRunningTime="2026-11-02 12:26:40.000059611 +0000 UTC m=+39.013289665"
Nov 02 12:26:40.963978 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:40.963889    2564 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f" event={"ID":"e51414af-06a2-416d-9656-871c483ba22a","Type":"ContainerStarted","Data":"3a142b439900a17c2c755d854b8acda7fdee568520197f5039db55bbd8f7bca3"}
Nov 02 12:26:41.005654 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.005593    2564 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f" podStartSLOduration=1.5907544009999999 podStartE2EDuration="3.005576867s" totalImagesPullingTime="1.414822466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:41.005520578 +0000 UTC m=+40.018750610" watchObservedRunningTime="2026-11-02 12:26:41.005576867 +0000 UTC m=+40.018806900"
Nov 02 12:26:41.270997 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.270959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:41.271165 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.271144    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:41.271328 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:41.271308    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:41.271389 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:41.271378    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.271363963 +0000 UTC m=+44.284593987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:41.271765 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:41.271747    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:41.271857 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:41.271807    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.271791271 +0000 UTC m=+44.285021297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:41.703691 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.703622    2564 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:41.703691 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.703689    2564 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:41.704047 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:41.704034    2564 scope.go:122] "RemoveContainer" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"
Nov 02 12:26:41.704235 ip-10-0-135-160 kubenswrapper[2564]: E1102 12:26:41.704214    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:42.285643 ip-10-0-135-160 kubenswrapper[2564]: I1102 12:26:42.285517    2564 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:42.285687 ip-10-0-135-160 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 12:26:42.306205 ip-10-0-135-160 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 12:26:42.306390 ip-10-0-135-160 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 12:26:42.306440 ip-10-0-135-160 systemd[1]: kubelet.service: Consumed 3.011s CPU time.
Nov 02 12:26:42.321543 ip-10-0-135-160 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:26:42.407946 ip-10-0-135-160 kubenswrapper[4928]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:42.407946 ip-10-0-135-160 kubenswrapper[4928]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:26:42.407946 ip-10-0-135-160 kubenswrapper[4928]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:42.407946 ip-10-0-135-160 kubenswrapper[4928]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:42.410187 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410101    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.410187 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410183    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.410187 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410190    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410195    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410201    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410206    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410210    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410214    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410218    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410222    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410226    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410230    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410235    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410239    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410243    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410247    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410251    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410255    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410259    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410263    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410266    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410271    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.410412 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410275    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410279    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410296    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410301    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410305    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410309    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410313    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410317    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410323    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410328    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410332    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410337    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410341    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410345    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410350    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410355    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410359    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410363    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410367    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410372    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.411315 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410376    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410381    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410385    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410389    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410394    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410398    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410405    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410412    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410416    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410422    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410427    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410432    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410436    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410441    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410446    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410451    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410455    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410460    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.412224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410466    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410470    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410475    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410479    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410484    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410488    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410495    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410502    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410507    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410512    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410516    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410520    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410524    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410529    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410533    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410538    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410542    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410546    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410551    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.413062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410555    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410559    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410564    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410569    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410574    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410579    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410583    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410587    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410591    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410596    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410601    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410605    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410610    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410614    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410620    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410625    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410629    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410634    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410638    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.413952 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410642    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410647    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410651    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410655    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410659    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410663    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410668    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.410672    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411924    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411936    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411942    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411947    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411951    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411956    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411960    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411965    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411970    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411975    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411979    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411983    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.414814 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411988    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411992    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.411996    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412000    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412005    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412010    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412014    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412019    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412024    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412029    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412033    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412037    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412042    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412046    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412050    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412055    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412059    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412063    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412068    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412072    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.415789 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412077    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412081    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412085    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412090    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412094    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412098    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412103    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412110    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412116    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412122    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412127    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412131    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412136    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412141    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412146    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412150    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412154    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412159    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412164    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.416654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412168    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412173    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412178    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412182    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412188    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412192    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412197    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412201    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412206    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412210    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412215    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412219    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412223    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412227    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412232    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412236    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412240    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412244    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412249    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.417194 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412253    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412257    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412262    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412266    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412270    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412275    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412279    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412283    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412288    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412293    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412297    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412302    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412307    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412311    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412316    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412320    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412324    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412329    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412333    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412338    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.417671 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412342    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412347    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412352    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412356    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412360    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412364    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412369    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412373    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412377    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412383    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412390    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412395    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412400    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412404    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412409    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.412413    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413276    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413288    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413293    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.418221 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413298    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413302    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413307    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413311    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413316    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413321    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413326    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413330    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413335    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413339    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413343    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413348    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413352    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413356    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413360    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413365    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413369    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413374    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413378    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413382    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.418706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413386    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413390    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413395    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413399    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413403    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413408    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413412    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413417    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413421    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413425    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413429    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413434    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413438    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413442    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413446    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413451    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413455    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413459    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413464    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.419228 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413468    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413473    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413477    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413504    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413519    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413527    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413534    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413539    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413543    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413548    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413553    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413557    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413561    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413565    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413570    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413574    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413578    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413582    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413586    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.419705 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413591    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413596    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413600    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413604    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413608    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413613    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413617    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413621    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413625    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413631    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413635    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413639    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413643    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413649    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413653    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413657    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413662    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413666    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413670    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.420310 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413674    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413680    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413687    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413693    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413697    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413702    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413706    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413711    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413715    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413720    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413728    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413734    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413739    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413743    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413747    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413751    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413756    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413760    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413764    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413768    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.420784 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413772    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413777    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413781    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413785    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413790    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.413794    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413936    4928 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413951    4928 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413960    4928 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413967    4928 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413974    4928 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413979    4928 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413987    4928 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413994    4928 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.413999    4928 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414005    4928 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414011    4928 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414017    4928 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414022    4928 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414027    4928 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414031    4928 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414036    4928 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414045    4928 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:26:42.421403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414050    4928 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414057    4928 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414062    4928 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414067    4928 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414072    4928 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414078    4928 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414085    4928 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414090    4928 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414095    4928 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414100    4928 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414106    4928 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414110    4928 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414115    4928 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414121    4928 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414125    4928 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414132    4928 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414137    4928 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414143    4928 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414148    4928 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414152    4928 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414157    4928 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414164    4928 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414169    4928 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414175    4928 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:26:42.422314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414180    4928 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414185    4928 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414192    4928 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414196    4928 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414201    4928 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414206    4928 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414211    4928 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414218    4928 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414223    4928 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414227    4928 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414232    4928 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414237    4928 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414241    4928 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414248    4928 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414253    4928 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414258    4928 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414263    4928 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414268    4928 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414273    4928 flags.go:64] FLAG: --help="false"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414278    4928 flags.go:64] FLAG: --hostname-override="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414283    4928 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414288    4928 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414293    4928 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414299    4928 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:26:42.422980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414307    4928 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414312    4928 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414319    4928 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414324    4928 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414328    4928 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414333    4928 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414338    4928 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414343    4928 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414349    4928 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414353    4928 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414358    4928 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414363    4928 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414367    4928 flags.go:64] FLAG: --lock-file=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414372    4928 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414376    4928 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414383    4928 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414392    4928 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414397    4928 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414402    4928 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414407    4928 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414411    4928 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414417    4928 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414421    4928 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414426    4928 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414434    4928 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:26:42.423616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414439    4928 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414445    4928 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414449    4928 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414455    4928 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414459    4928 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414464    4928 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414469    4928 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414476    4928 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414481    4928 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414492    4928 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414498    4928 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414503    4928 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414508    4928 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414513    4928 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414518    4928 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414523    4928 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414527    4928 flags.go:64] FLAG: --port="10250"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414532    4928 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414537    4928 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f658221cc576a613"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414542    4928 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414547    4928 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414552    4928 flags.go:64] FLAG: --register-node="true"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414559    4928 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414565    4928 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414570    4928 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414575    4928 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:26:42.424344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414579    4928 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414585    4928 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414590    4928 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414595    4928 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414600    4928 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414605    4928 flags.go:64] FLAG: --runonce="false"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414610    4928 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414615    4928 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414620    4928 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414625    4928 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414629    4928 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414635    4928 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414640    4928 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414648    4928 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414653    4928 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414658    4928 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414663    4928 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414668    4928 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414673    4928 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414678    4928 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414683    4928 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414692    4928 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414697    4928 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414702    4928 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414708    4928 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:26:42.425045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414713    4928 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414718    4928 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414722    4928 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414728    4928 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414733    4928 flags.go:64] FLAG: --v="2"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414740    4928 flags.go:64] FLAG: --version="false"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414747    4928 flags.go:64] FLAG: --vmodule=""
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414753    4928 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.414758    4928 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.414986    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.414995    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415000    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415005    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415009    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415014    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415020    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415026    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415046    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415050    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415055    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415062    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.425839 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415066    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415070    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415075    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415079    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415085    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415089    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415094    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415098    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415102    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415107    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415111    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415115    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415119    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415124    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415130    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415135    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415139    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415143    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415147    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415151    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.426470 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415156    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415160    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415165    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415169    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415174    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415178    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415182    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415186    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415190    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415194    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415199    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415209    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415214    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415218    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415223    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415227    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415231    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415236    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415240    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415244    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.427170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415249    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415253    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415257    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415262    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415266    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415271    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415276    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415281    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415287    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415293    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415298    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415303    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415307    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415332    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415338    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415344    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415349    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415354    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415359    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.427699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415363    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415367    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415371    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415376    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415382    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415387    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415391    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415396    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415400    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415405    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415409    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415414    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415419    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415423    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415427    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415431    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415436    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415440    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415445    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.428220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415451    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415455    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415460    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415464    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415469    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415474    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415478    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415483    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415487    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415491    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415496    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415500    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415504    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415509    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415513    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415517    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.428781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.415558    4928 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415614    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415622    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415627    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415631    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415636    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415641    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415645    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415650    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415655    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415659    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415663    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415668    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415672    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415676    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415681    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415685    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415692    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415696    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415700    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.429278 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415705    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415709    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415714    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415718    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415722    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415726    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415731    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415735    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415740    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415744    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415748    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415753    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415757    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415765    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415769    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415773    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415778    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415782    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415786    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415790    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.429815 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415794    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415798    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415803    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415807    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415812    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415816    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415820    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415843    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415851    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415860    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415865    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415870    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415874    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415878    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415882    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415886    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415890    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415895    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415900    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.430419 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415904    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415908    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415912    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415917    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415921    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415925    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415931    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415935    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415939    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415944    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415948    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415953    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415957    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415961    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415966    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415970    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415975    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415980    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415984    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415989    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.430898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415993    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.415997    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416003    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416007    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416012    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416016    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416020    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416025    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416029    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416034    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416038    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416043    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416047    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416051    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416055    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416059    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416063    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416068    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416074    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.431428 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416079    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416083    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416087    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416092    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416098    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416105    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416109    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416115    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.416120    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.420223    4928 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.420240    4928 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.420259    4928 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.420264    4928 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420340    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420345    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420349    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420352    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420355    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420358    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.432009 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420360    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420364    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420368    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420370    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420373    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420376    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420385    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420389    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420392    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420394    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420397    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420399    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420402    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420405    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420407    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420410    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420413    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420415    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420418    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.432472 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420420    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420423    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420425    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420428    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420431    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420440    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420443    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420446    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420449    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420451    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420454    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420456    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420459    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420462    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420465    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420468    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420470    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420473    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420476    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420478    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.432961 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420487    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420490    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420492    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420495    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420498    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420500    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420503    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420506    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420518    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420521    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420523    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420526    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420528    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420531    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420534    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420536    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420539    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420543    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420545    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.433436 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420548    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420550    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420553    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420555    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420558    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420561    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420563    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420567    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420569    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420572    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420575    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420577    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420580    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420582    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420592    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420595    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420598    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420602    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420605    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.433933 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420608    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420611    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420614    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420617    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420619    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420622    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420625    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420628    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420630    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420633    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420636    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420639    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420642    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420646    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420651    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420655    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420659    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420662    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420667    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420671    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.434434 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420674    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420677    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420680    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.420709    4928 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420918    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420923    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420927    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420929    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420939    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420942    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420944    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420947    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420949    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420952    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420955    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.435032 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420957    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420959    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420962    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420965    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420968    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420970    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420973    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420976    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420978    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420981    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420984    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420987    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420989    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420992    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420994    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.420997    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421000    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421002    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421005    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421007    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421010    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421012    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421015    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421017    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421020    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421023    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421032    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421036    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421040    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421043    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421045    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421048    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421052    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421056    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421059    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421062    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421065    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421068    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421071    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.436217 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421074    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421076    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421079    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421082    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421084    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421087    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421090    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421092    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421095    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421097    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421100    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421102    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421105    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421107    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421110    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421112    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421115    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421117    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421120    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421122    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.436840 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421161    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421319    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421324    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421329    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421334    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421341    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421346    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421351    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421356    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421362    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421367    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421373    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421386    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421391    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421396    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421400    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421405    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421411    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421417    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.437465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421425    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421432    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421439    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421445    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421450    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421455    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421467    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421472    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421477    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421483    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421487    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421494    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421499    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421503    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421543    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421550    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.438169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421555    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.421620    4928 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421803    4928 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421811    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421816    4928 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421842    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421846    4928 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421852    4928 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421857    4928 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421862    4928 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421867    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421872    4928 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421876    4928 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421881    4928 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421886    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:42.438797 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421891    4928 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421896    4928 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421905    4928 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421910    4928 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421915    4928 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421920    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421925    4928 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421929    4928 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421934    4928 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421939    4928 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421944    4928 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421949    4928 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421954    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421958    4928 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421963    4928 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.421973    4928 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422006    4928 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422011    4928 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422016    4928 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422021    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:42.439425 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422026    4928 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422031    4928 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422036    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422041    4928 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422046    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422050    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422055    4928 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422090    4928 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422095    4928 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422100    4928 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422105    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422139    4928 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422146    4928 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422154    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422159    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422165    4928 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422170    4928 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422176    4928 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422183    4928 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422193    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:42.440179 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422199    4928 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422203    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422208    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422216    4928 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422222    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422232    4928 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422236    4928 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422241    4928 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422389    4928 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422402    4928 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422406    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422410    4928 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422413    4928 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422415    4928 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422418    4928 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422421    4928 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422424    4928 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422426    4928 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422429    4928 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422432    4928 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:42.440724 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422434    4928 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422437    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422440    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422443    4928 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422446    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422449    4928 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422451    4928 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422454    4928 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422457    4928 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422460    4928 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422462    4928 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422465    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422468    4928 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422471    4928 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422473    4928 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422476    4928 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422479    4928 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422482    4928 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422484    4928 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:42.441237 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422487    4928 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422490    4928 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422495    4928 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422498    4928 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422501    4928 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422504    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422506    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422512    4928 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422515    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422518    4928 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422521    4928 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422524    4928 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422527    4928 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:42.422529    4928 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.422598    4928 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.423647    4928 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.423697    4928 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.423894    4928 server.go:1014] "Starting client certificate rotation"
Nov 02 12:26:42.441725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.424039    4928 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.424109    4928 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:01 +0000 UTC" deadline="2028-06-09 19:12:20.828626433 +0000 UTC"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.424149    4928 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14046h45m38.404481278s"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.424315    4928 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.425382    4928 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.426590    4928 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.428264    4928 log.go:25] "Validated CRI v1 image API"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.429033    4928 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:26:42.442263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.430862    4928 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8568abb0-7df4-484e-96d5-9ac2e0a1c555:/dev/nvme0n1p4 b478e783-b0a2-42a4-96ee-1384324d3662:/dev/nvme0n1p3]
Nov 02 12:26:42.442497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.430883    4928 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/561894cb2ed866884ca46542463138c4433f924191afb90055696f3dffd53f2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/561894cb2ed866884ca46542463138c4433f924191afb90055696f3dffd53f2a/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7c2ac9013838830d1ada98474e0ec0a5f8ece0aee142fab629a4ff6b617816b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7c2ac9013838830d1ada98474e0ec0a5f8ece0aee142fab629a4ff6b617816b/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be565a3159839d56a4ddfc70b5cd2fa0082d5d653a239069da1ead0cf57d4457/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be565a3159839d56a4ddfc70b5cd2fa0082d5d653a239069da1ead0cf57d4457/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04856dfa-3c20-45b0-bc5f-f68896f87c51/volumes/kubernetes.io~projected/kube-api-access-bpkwq:{mountpoint:/var/lib/kubelet/pods/04856dfa-3c20-45b0-bc5f-f68896f87c51/volumes/kubernetes.io~projected/kube-api-access-bpkwq major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~projected/kube-api-access-m68d7:{mountpoint:/var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~projected/kube-api-access-m68d7 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~projected/kube-api-access-dw9xd:{mountpoint:/var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~projected/kube-api-access-dw9xd major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e8b0b9d-80b8-460e-98d3-189c240cd20d/volumes/kubernetes.io~projected/kube-api-access-cwbqt:{mountpoint:/var/lib/kubelet/pods/0e8b0b9d-80b8-460e-98d3-189c240cd20d/volumes/kubernetes.io~projected/kube-api-access-cwbqt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~projected/kube-api-access-prqrj:{mountpoint:/var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~projected/kube-api-access-prqrj major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d0715df-bd6b-45ec-b51a-fb79504da0e5/volumes/kubernetes.io~projected/kube-api-access-m6ppz:{mountpoint:/var/lib/kubelet/pods/2d0715df-bd6b-45ec-b51a-fb79504da0e5/volumes/kubernetes.io~projected/kube-api-access-m6ppz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/kube-api-access-72k2k:{mountpoint:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/kube-api-access-72k2k major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3292ca18-a18b-404e-bf25-e2d7cfb9d383/volumes/kubernetes.io~projected/kube-api-access-n7v44:{mountpoint:/var/lib/kubelet/pods/3292ca18-a18b-404e-bf25-e2d7cfb9d383/volumes/kubernetes.io~projected/kube-api-access-n7v44 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab/volumes/kubernetes.io~projected/kube-api-access-mn7nf:{mountpoint:/var/lib/kubelet/pods/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab/volumes/kubernetes.io~projected/kube-api-access-mn7nf major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/kube-api-access-nzhxn:{mountpoint:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/kube-api-access-nzhxn major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66807b2c-d92d-48aa-815d-4b437efae8eb/volumes/kubernetes.io~projected/kube-api-access-pxrgr:{mountpoint:/var/lib/kubelet/pods/66807b2c-d92d-48aa-815d-4b437efae8eb/volumes/kubernetes.io~projected/kube-api-access-pxrgr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69559f09-3ce2-435a-a738-1d6892629b7d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/69559f09-3ce2-435a-a738-1d6892629b7d/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~projected/kube-api-access-bnr47:{mountpoint:/var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~projected/kube-api-access-bnr47 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cf5eb1e-e80f-40d5-9367-4af0c929723f/volumes/kubernetes.io~projected/kube-api-access-hhwpv:{mountpoint:/var/lib/kubelet/pods/7cf5eb1e-e80f-40d5-9367-4af0c929723f/volumes/kubernetes.io~projected/kube-api-access-hhwpv major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e3c9141-625c-449d-8b94-3aa76c9a5d04/volumes/kubernetes.io~projected/kube-api-access-zbnl7:{mountpoint:/var/lib/kubelet/pods/7e3c9141-625c-449d-8b94-3aa76c9a5d04/volumes/kubernetes.io~projected/kube-api-access-zbnl7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8250eede-bd12-46d3-a569-6a242bbfd566/volumes/kubernetes.io~projected/kube-api-access-jcvpt:{mountpoint:/var/lib/kubelet/pods/8250eede-bd12-46d3-a569-6a242bbfd566/volumes/kubernetes.io~projected/kube-api-access-jcvpt major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~projected/kube-api-access-hh6vg:{mountpoint:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~projected/kube-api-access-hh6vg major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~projected/kube-api-access-qqrnh:{mountpoint:/var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~projected/kube-api-access-qqrnh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9939d8b4-1181-463b-a465-13da054067b6/volumes/kubernetes.io~projected/kube-api-access-jc8h2:{mountpoint:/var/lib/kubelet/pods/9939d8b4-1181-463b-a465-13da054067b6/volumes/kubernetes.io~projected/kube-api-access-jc8h2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a12625f-91bf-4fc3-a398-eaa065d811de/volumes/kubernetes.io~projected/kube-api-access-smkfd:{mountpoint:/var/lib/kubelet/pods/9a12625f-91bf-4fc3-a398-eaa065d811de/volumes/kubernetes.io~projected/kube-api-access-smkfd major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a76295dc-1849-4d96-8940-cdc1de45551e/volumes/kubernetes.io~projected/kube-api-access-gblvc:{mountpoint:/var/lib/kubelet/pods/a76295dc-1849-4d96-8940-cdc1de45551e/volumes/kubernetes.io~projected/kube-api-access-gblvc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7d37dd0-8676-4b17-81a4-201865d84b89/volumes/kubernetes.io~projected/kube-api-access-7vn2h:{mountpoint:/var/lib/kubelet/pods/a7d37dd0-8676-4b17-81a4-201865d84b89/volumes/kubernetes.io~projected/kube-api-access-7vn2h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5b28cf8-a271-41c8-a765-811b45bfeeb0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c5b28cf8-a271-41c8-a765-811b45bfeeb0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~projected/kube-api-access-4zkmn:{mountpoint:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~projected/kube-api-access-4zkmn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce9a1148-d94a-47ad-8b2d-147af29a5797/volumes/kubernetes.io~projected/kube-api-access-76chf:{mountpoint:/var/lib/kubelet/pods/ce9a1148-d94a-47ad-8b2d-147af29a5797/volumes/kubernetes.io~projected/kube-api-access-76chf major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceb68925-753a-4c72-8d6d-01b1e2550c6a/volumes/kubernetes.io~projected/kube-api-access-9x6vh:{mountpoint:/var/lib/kubelet/pods/ceb68925-753a-4c72-8d6d-01b1e2550c6a/volumes/kubernetes.io~projected/kube-api-access-9x6vh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef/volumes/kubernetes.io~projected/kube-api-access-bqs5x:{mountpoint:/var/lib/kubelet/pods/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef/volumes/kubernetes.io~projected/kube-api-access-bqs5x major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e51414af-06a2-416d-9656-871c483ba22a/volumes/kubernetes.io~projected/kube-api-access-5qvx4:{mountpoint:/var/lib/kubelet/pods/e51414af-06a2-416d-9656-871c483ba22a/volumes/kubernetes.io~projected/kube-api-access-5qvx4 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~projected/kube-api-access-rk6wt:{mountpoint:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~projected/kube-api-access-rk6wt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2849f75-3d38-49d2-8372-fd7776ddf12e/volumes/kubernetes.io~projected/kube-api-access-s2qt9:{mountpoint:/var/lib/kubelet/pods/f2849f75-3d38-49d2-8372-fd7776ddf12e/volumes/kubernetes.io~projected/kube-api-access-s2qt9 major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1f3e2ed0a44a23ab5b6a649449a4122e6085dd14ebe4a8cf611cb771cf7b1ef1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/397836fe08282a9ebbf3d9b1f6effd3e39ca249586e9d81c39e96b5822b14565/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7e5c1bd1e51725ce981fae8cabfa3eefdbf8d28ded2991aba15e8a846e513856/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4ed021c6a4222d6f7e27ec530c7cf995f558e0b99ba0d3602c38fc941c2d49ad/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f4382b81fe6ccfe4be38e211f8179aa3c0e63a7ed92ebf2dfdc399954a98cc97/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9b788752be4752050676a804009e3ae91c02dd2367e225745515c13f14d5128d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ef0bc9127edb00176a8fa5a40c6d5cf302549c05f51ce38907e5c43bc7d9bf03/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1f207d4510beb801f3d0a15f717dacd37682f58e1c7ce04334e85ec8106691f4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/95f6f8ed869a672246bdcc8841faaf55c4a58502532c7cf645ffbc267dcbd8d5/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d7ba06fa24ad706c6c6cfbdee7a2d44c3bab4b34ce2057150907d25065b0feee/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/2a9871bf13774d127bd683186c923d68a48460eb192dc2c2bd9769a8b257ed3c/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1893b1980ba2f010f0f522559df4fe60beae238fe49d1db75ad94dd1cc9a9062/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5bcec18ab2a3e7623a202d3f7671be385a9850e61bede4a0efb0cebce08911ee/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4f1c0f0e160c586b51051061867b768215cb1a91a3d5f6f4bbdd6e8cb9ab6579/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9fe86afaaa2320d7277c890ce9b77f0b5702be832d328572c6a6f944ee8aa379/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/4ec9acfe74a45714ccf207f8b4409312b45b0706791d6cf1a741b1f1fbf69188/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/3955387309d77d8d493d5c0e751a3c9d4414e1a48ad20350ffdc700b069be32a/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/042bdf7c26b2871b4267c09b28c16a07e4a9964d66a17dab57d8013c03e8e5fa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5d937bb304103f5675d0a3022450ed075b17ef7bf48c06416d852f9e522b4734/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fc873a2cbcea2c0d76ea2aaaab31b633bcfd04dfd85a4936283df5e57fe30bcc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8c45a1210a1833c19d5b0a1f383f311bd6a59b0f19365095f4c5c712a7561a45/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/324d8650809d6a223b52df384467f2de17f108fe487fd857f5118c8dbcb9878f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d260406987ef33ba611718a24c2e1c932c93f4d673e7ab1ec42f638bf2f0cbfc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c7c48129f3f1fe963765996e9911408f6bb7a432f0f754d57cb9de856d09a2f6/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/6a3c720db9b4b0108344d6179c97911080766989b661430dcdf800e5debec94f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a5d489381501b4759a54c718e889c00e9d0ecc9a199945e11a920c672126633f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/fff080566146215d986443e6d2add7da9cc84c1616f7cb7dc691cbc84fb6cf69/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/52cfce1e65a3664f48a3be0f89ea0b0371d466f0d29f128608e27656ac13ca2c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f8590ef63d09589cf1cdaed717d6edc7685b9885992eea6b889a11dca95301cc/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e5f9f2471404d43d670e5902077b82ca8685957f76f72241c13c0ba99c5a6780/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/263434b92613c6a1d2aeefcfce75a1ae9a5da6cbc7c1e10f5362c1c5d4ec8eb8/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/e2a9f312a0a01362855448d6f81c270550a209b56f7e5022f0d4705b1ebde407/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/598b6c536f615a45a3f16739bf62a360f9c72e604e12221b3c0b63705b49394c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/d674672e1ddc386a350a1aee9ebf84fde38342cd05aeca0d9136b305e90e8af2/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/31883425dd1457fec45c95f1f230f72ecf7f7ca6a449bd2bfdef76c0cbbcbbc6/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/605de47b986c41372d341a3976739fdfb3b054b61e5b9db8a817940435b7b326/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a4d279ee6c689dd138254de7f9288b7790696a3507e776f89e837656d2403c2c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/13cbee83d54101d21ba3797a86a92ed1b91c691d59ebff6572dc892283d316ae/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/87d9ba58d96d9c4466cdd5cf907562f72afa9d9f1ed7eac8017b37756d27c090/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0e6e5a4e7961ff0f745e6219588966ad8efbe1a3f7a09772bf46aa331bdfb61f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/9d8c7083debedaedd844a53c4ccaff63ec90ad571b63e3266a5f9a5f8ddd57ea/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/c873199bb557cd5e043370567d46d0a0f7f8dd1df0c5f3911f55f8c226f83a67/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/7170d5081fc42350c272deecf3c1242b94c4f788db838d913b38c34d714868ce/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/56b76aae7488a531255fa1126d910a94adda998aaecf14a8ef9babd2a6dbf0b9/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/413f623541227b86cbb57d18d7cec90a08440b2b7ff8546774efbe0aab121c36/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/84fd2167e812b6c7490e801f01e31312af8bde82bb503f6a58854bc37ef9ed31/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/be3100aa05789b8173e36115be616e60e3391a89b0f2f8e60dfc93697e2f17ca/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/030316dac62183ef78ee7e7aa6a8864fe3f43ddb9aa9234451bb0cedbb35d055/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/80bc9164a017873f94af7cbfa419e581a21631461d809e802ae8e28875daf9cb/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c24d5164a2cf6209b2fe3e790b550a6678061592ac256c29aa2a78de33f6f7bf/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9d3558e8f357cbb7cad28d484d7abadfc26fecbb16d0cd78a6fa9179038cc4cc/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/98434502e357f9303636f6563d513beeb6dcdc71c0cf5927edf23f796092dd8e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e0c55b742d7462bc99ffa984400e359b33d1382cb1de680bfd1c6f7c7a388486/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/246b93c4152f49935b1fe81f0454330f56447bc2fd45b289e97cae213829b1b9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/9b0e3ffbf03ac81373ac89145e52a6a40beb8956125b8feb2ebb06ad97567ef7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4754eb58e299d9afc9d838373381d75bee2e8022f950e735f3d090ab2d082a3e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.440750    4928 manager.go:217] Machine: {Timestamp:2026-11-02 12:26:42.439870146 +0000 UTC m=+0.088924097 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104744 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2571b7ba6e5a25fffcd6b18ffa14cf SystemUUID:ec2571b7-ba6e-5a25-fffc-d6b18ffa14cf BootID:00570ddc-d6f1-459f-8864-9a976f362aa4 Filesystems:[{Device:/run/containers/storage/overlay-containers/9850c5c7199707fd59b5bd2502a9f64ff0bf1c24447b136a885985391c748c82/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2849f75-3d38-49d2-8372-fd7776ddf12e/volumes/kubernetes.io~projected/kube-api-access-s2qt9 DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e3c9141-625c-449d-8b94-3aa76c9a5d04/volumes/kubernetes.io~projected/kube-api-access-zbnl7 DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e8b0b9d-80b8-460e-98d3-189c240cd20d/volumes/kubernetes.io~projected/kube-api-access-cwbqt DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b5769a95bf0aa175f0aca7cf7431ce9347c5c6d2b3ff3e44f56d75c55d1abd4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/145895484b92436ed2db8b9f10fabf27203c02d92c999a33519683a36a57e141/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151/volumes/kubernetes.io~projected/kube-api-access-dw9xd DeviceMajor:0 DeviceMinor:226 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~projected/kube-api-access-prqrj DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c80f5cccb7e6439af080d8649f43dc6aeadfbf3c190b305f9fd11fe179524c92/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9b61be719ba1b254df392b71fecf0dcfc7f996208cfd18c2c29846161ff1bfc/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00981feb0c12fa6430b492ef3d8473d5068959affe64e31e99b9ca9a021d1b75/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~projected/kube-api-access-bnr47 DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce9a1148-d94a-47ad-8b2d-147af29a5797/volumes/kubernetes.io~projected/kube-api-access-76chf DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~projected/kube-api-access-rk6wt DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9bd9a0607dab0cb7f738e51f8b27afceb8f2e1814e5643d32dc5f0270566651/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9939d8b4-1181-463b-a465-13da054067b6/volumes/kubernetes.io~projected/kube-api-access-jc8h2 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3292ca18-a18b-404e-bf25-e2d7cfb9d383/volumes/kubernetes.io~projected/kube-api-access-n7v44 DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9a12625f-91bf-4fc3-a398-eaa065d811de/volumes/kubernetes.io~projected/kube-api-access-smkfd DeviceMajor:0 DeviceMinor:312 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~projected/kube-api-access-nzhxn DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bac3cc2f03e3014edd9e42a5373cb0c51cc0b70eb0d5aa3ab3c62a5f36bb9f4/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10ae75b6-203e-4d60-b728-412645e0a4a8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7d37dd0-8676-4b17-81a4-201865d84b89/volumes/kubernetes.io~projected/kube-api-access-7vn2h DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ceb68925-753a-4c72-8d6d-01b1e2550c6a/volumes/kubernetes.io~projected/kube-api-access-9x6vh DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c7d1690824cde0b3221ca02fda2391c9dc2882bd6bbf7813d1aa8c1b9b0c510/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19360505b8ecf80dcc9faba58d81892d6e9ded138f93592b7c06031377bde752/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2d0715df-bd6b-45ec-b51a-fb79504da0e5/volumes/kubernetes.io~projected/kube-api-access-m6ppz DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5b28cf8-a271-41c8-a765-811b45bfeeb0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8250eede-bd12-46d3-a569-6a242bbfd566/volumes/kubernetes.io~projected/kube-api-access-jcvpt DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e642ca0bfb08bade02af963a1a7658bca824f91a277ad12957fc13d7e91703e7/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/086665ff-9fcc-48af-982e-59bb0239605b/volumes/kubernetes.io~projected/kube-api-access-m68d7 DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7c2ac9013838830d1ada98474e0ec0a5f8ece0aee142fab629a4ff6b617816b/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/700faa780a782479f1c3072c314cd61d65413304225d9c5c0e31bfcf0d0539ad/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cf5eb1e-e80f-40d5-9367-4af0c929723f/volumes/kubernetes.io~projected/kube-api-access-hhwpv DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/69559f09-3ce2-435a-a738-1d6892629b7d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ae63a0f592afbfbeca5837cca548f5b317f1c723b923167ab8dc56f9a5e35e0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04856dfa-3c20-45b0-bc5f-f68896f87c51/volumes/kubernetes.io~projected/kube-api-access-bpkwq DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d1cf3e551ec356ab0f6fde92d1b953fd03fd74ce687c7ab00265bd552eff1b3/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/kube-api-access-72k2k DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~projected/kube-api-access-qqrnh DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/561894cb2ed866884ca46542463138c4433f924191afb90055696f3dffd53f2a/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3dee460a44b1b1d069177182ba7d088650c3912274aba4bd982007748d9f2db/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab/volumes/kubernetes.io~projected/kube-api-access-mn7nf DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a76295dc-1849-4d96-8940-cdc1de45551e/volumes/kubernetes.io~projected/kube-api-access-gblvc DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e331122052a7697bae8605c5bcd97a4f59c1693066ab21c5f95fd178e70934ca/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e51414af-06a2-416d-9656-871c483ba22a/volumes/kubernetes.io~projected/kube-api-access-5qvx4 DeviceMajor:0 DeviceMinor:325 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/66807b2c-d92d-48aa-815d-4b437efae8eb/volumes/kubernetes.io~projected/kube-api-access-pxrgr DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae388d9e7671ddee7a89f2d6ecde3c972215781d4c592a7e32802e95923ed918/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6da4be61-ac40-4531-9357-e05f84219250/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0b7f5d-71cf-4491-a17b-63cb47bb6441/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8740f87ea76c9cc2794293d11f1ff7e2905c2ce7c76962c0f3ced20ddd35764/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~projected/kube-api-access-4zkmn DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e2649ce-6913-40b6-a47a-cd71b5208dd3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ee3214-5327-4181-8c06-5a9c350ed235/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef/volumes/kubernetes.io~projected/kube-api-access-bqs5x DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82d00d41-1c4a-450f-9a27-8571dc21bf21/volumes/kubernetes.io~projected/kube-api-access-hh6vg DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be565a3159839d56a4ddfc70b5cd2fa0082d5d653a239069da1ead0cf57d4457/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00981feb0c12fa6 MacAddress:9a:5e:86:20:ac:fa Speed:10000 Mtu:8901} {Name:0b5769a95bf0aa1 MacAddress:4a:c9:b5:71:28:d7 Speed:10000 Mtu:8901} {Name:19360505b8ecf80 MacAddress:5e:fb:1f:19:96:d3 Speed:10000 Mtu:8901} {Name:2bac3cc2f03e301 MacAddress:76:f0:17:7e:3e:aa Speed:10000 Mtu:8901} {Name:561894cb2ed8668 MacAddress:8e:98:6c:7e:3f:c1 Speed:10000 Mtu:8901} {Name:5d1cf3e551ec356 MacAddress:72:c8:98:5e:be:1e Speed:10000 Mtu:8901} {Name:a7c2ac901383883 MacAddress:aa:16:5b:c6:48:34 Speed:10000 Mtu:8901} {Name:a8740f87ea76c9c MacAddress:b2:e8:8e:5e:15:b3 Speed:10000 Mtu:8901} {Name:b3dee460a44b1b1 MacAddress:96:0a:9c:5b:d1:01 Speed:10000 Mtu:8901} {Name:be565a3159839d5 MacAddress:6a:7d:16:81:cd:c4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:42:08:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:24:01:2a:56:31 Speed:0 Mtu:8901} {Name:d9b61be719ba1b2 MacAddress:4e:57:73:27:39:78 Speed:10000 Mtu:8901} {Name:e331122052a7697 MacAddress:da:b6:d1:d2:24:d0 Speed:10000 Mtu:8901} {Name:e642ca0bfb08bad MacAddress:b6:9d:f2:68:31:24 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:42:08:51 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ba:e8:de:05:1e:b8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:ea:23:0e:b6:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441133    4928 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441181    4928 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441505    4928 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441540    4928 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441676    4928 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441685    4928 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441698    4928 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441714    4928 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441733    4928 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.441904    4928 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442010    4928 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442044    4928 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442059    4928 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442069    4928 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442078    4928 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:26:42.442734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.442096    4928 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:26:42.443359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.443123    4928 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:26:42.443359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.443138    4928 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:42.443705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.443685    4928 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:26:42.446036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446022    4928 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:26:42.446036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446036    4928 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:42.446144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446134    4928 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 12:26:42.446348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446336    4928 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:26:42.446395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446386    4928 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:26:42.446510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446498    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446521    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446530    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446538    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446546    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446552    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446557    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:26:42.446561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446563    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446571    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446577    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446586    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446596    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446638    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:26:42.446816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.446644    4928 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:26:42.450342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.450327    4928 server.go:1263] "Started kubelet"
Nov 02 12:26:42.450802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.450398    4928 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:26:42.450802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.450434    4928 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:26:42.450802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.450529    4928 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:26:42.451053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.450856    4928 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:26:42.450930 ip-10-0-135-160 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:26:42.452364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.452347    4928 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456157    4928 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456235    4928 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456272    4928 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456285    4928 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456278    4928 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:03 +0000 UTC" deadline="2028-08-18 08:39:25.202898024 +0000 UTC"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456300    4928 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15716h12m42.746601628s"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.456309    4928 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-160.ec2.internal\" not found"
Nov 02 12:26:42.456454 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.456349    4928 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:26:42.457399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457378    4928 factory.go:55] Registering systemd factory
Nov 02 12:26:42.457399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457400    4928 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457730    4928 factory.go:153] Registering CRI-O factory
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457748    4928 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457845    4928 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457868    4928 factory.go:103] Registering Raw factory
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.457884    4928 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:26:42.458970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.458235    4928 manager.go:319] Starting recovery of all containers
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.458985    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e2649ce-6913-40b6-a47a-cd71b5208dd3" volumeName="kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459028    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3292ca18-a18b-404e-bf25-e2d7cfb9d383" volumeName="kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459039    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459047    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0715df-bd6b-45ec-b51a-fb79504da0e5" volumeName="kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459057    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a76295dc-1849-4d96-8940-cdc1de45551e" volumeName="kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459065    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04856dfa-3c20-45b0-bc5f-f68896f87c51" volumeName="kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459073    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459080    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3c9141-625c-449d-8b94-3aa76c9a5d04" volumeName="kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459097    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5b28cf8-a271-41c8-a765-811b45bfeeb0" volumeName="kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459106    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceb68925-753a-4c72-8d6d-01b1e2550c6a" volumeName="kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459115    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459124    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459131    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459139    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459148    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceb68925-753a-4c72-8d6d-01b1e2550c6a" volumeName="kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459157    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0b7f5d-71cf-4491-a17b-63cb47bb6441" volumeName="kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459165    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66807b2c-d92d-48aa-815d-4b437efae8eb" volumeName="kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459173    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69559f09-3ce2-435a-a738-1d6892629b7d" volumeName="kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459180    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459187    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8250eede-bd12-46d3-a569-6a242bbfd566" volumeName="kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459194    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459201    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0715df-bd6b-45ec-b51a-fb79504da0e5" volumeName="kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459209    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10ae75b6-203e-4d60-b728-412645e0a4a8" volumeName="kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459216    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10ae75b6-203e-4d60-b728-412645e0a4a8" volumeName="kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459236    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0715df-bd6b-45ec-b51a-fb79504da0e5" volumeName="kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459244    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a76295dc-1849-4d96-8940-cdc1de45551e" volumeName="kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459252    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151" volumeName="kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459259    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce9a1148-d94a-47ad-8b2d-147af29a5797" volumeName="kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459270    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e51414af-06a2-416d-9656-871c483ba22a" volumeName="kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459278    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459285    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e586231-9b95-4114-82c2-602480b1e3c9" volumeName="kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459292    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459300    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf5eb1e-e80f-40d5-9367-4af0c929723f" volumeName="kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459309    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3c9141-625c-449d-8b94-3aa76c9a5d04" volumeName="kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459317    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d00d41-1c4a-450f-9a27-8571dc21bf21" volumeName="kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459325    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a12625f-91bf-4fc3-a398-eaa065d811de" volumeName="kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459332    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a76295dc-1849-4d96-8940-cdc1de45551e" volumeName="kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459340    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d0715df-bd6b-45ec-b51a-fb79504da0e5" volumeName="kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz" seLinuxMountContext=""
Nov 02 12:26:42.459322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459348    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459355    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459365    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459372    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459383    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459391    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a12625f-91bf-4fc3-a398-eaa065d811de" volumeName="kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459398    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0b7f5d-71cf-4491-a17b-63cb47bb6441" volumeName="kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459406    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9939d8b4-1181-463b-a465-13da054067b6" volumeName="kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459414    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d37dd0-8676-4b17-81a4-201865d84b89" volumeName="kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459422    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151" volumeName="kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459430    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10ae75b6-203e-4d60-b728-412645e0a4a8" volumeName="kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459437    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69559f09-3ce2-435a-a738-1d6892629b7d" volumeName="kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459444    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459452    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e2649ce-6913-40b6-a47a-cd71b5208dd3" volumeName="kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459464    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0b7f5d-71cf-4491-a17b-63cb47bb6441" volumeName="kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459472    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="086665ff-9fcc-48af-982e-59bb0239605b" volumeName="kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459479    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="086665ff-9fcc-48af-982e-59bb0239605b" volumeName="kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459489    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0b7f5d-71cf-4491-a17b-63cb47bb6441" volumeName="kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459510    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459518    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c4cc29-c98d-4fd9-b6d8-8090d72a11ab" volumeName="kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459526    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e2649ce-6913-40b6-a47a-cd71b5208dd3" volumeName="kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459540    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9939d8b4-1181-463b-a465-13da054067b6" volumeName="kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459547    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ee3214-5327-4181-8c06-5a9c350ed235" volumeName="kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459562    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ee3214-5327-4181-8c06-5a9c350ed235" volumeName="kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459570    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151" volumeName="kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459577    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e8b0b9d-80b8-460e-98d3-189c240cd20d" volumeName="kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459585    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3001af94-ab6c-4995-b400-edecd5c02f59" volumeName="kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459593    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e2649ce-6913-40b6-a47a-cd71b5208dd3" volumeName="kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459600    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d00d41-1c4a-450f-9a27-8571dc21bf21" volumeName="kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459608    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d37dd0-8676-4b17-81a4-201865d84b89" volumeName="kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459616    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ee3214-5327-4181-8c06-5a9c350ed235" volumeName="kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459623    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2849f75-3d38-49d2-8372-fd7776ddf12e" volumeName="kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459631    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532cb38b-3ea7-433c-88de-fb41dba50795" volumeName="kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459640    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf5eb1e-e80f-40d5-9367-4af0c929723f" volumeName="kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459648    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e8a733-2eca-4192-a6ab-67ccc40ff2ef" volumeName="kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459656    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0b7f5d-71cf-4491-a17b-63cb47bb6441" volumeName="kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459663    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3292ca18-a18b-404e-bf25-e2d7cfb9d383" volumeName="kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459670    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6da4be61-ac40-4531-9357-e05f84219250" volumeName="kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459678    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3c9141-625c-449d-8b94-3aa76c9a5d04" volumeName="kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459685    4928 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d00d41-1c4a-450f-9a27-8571dc21bf21" volumeName="kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth" seLinuxMountContext=""
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459693    4928 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:26:42.461399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.459699    4928 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:26:42.482487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.482453    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:42.482668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.482531    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:42.487315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.487266    4928 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:26:42.489118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.489087    4928 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:26:42.489118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.489113    4928 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:26:42.489351 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.489135    4928 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:26:42.489351 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.489168    4928 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 12:26:42.499938 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.499918    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:42.500050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.499946    4928 generic.go:356] "Generic (PLEG): container finished" podID="8e2649ce-6913-40b6-a47a-cd71b5208dd3" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b" exitCode=255
Nov 02 12:26:42.506936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506908    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="4d3f271442145d98b63d017877545c45dc1967f4a7044c49ed7cc37000e6a3f9" exitCode=0
Nov 02 12:26:42.506936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506930    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="eea4d9df0a8d78440235d95b0033f874e13f33e1cf461804c49460038d760086" exitCode=0
Nov 02 12:26:42.506936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506937    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="68f826b6c5789e33cae5932563fcc8265dee04a59f1c8d020150c6e07616d83a" exitCode=0
Nov 02 12:26:42.506936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506943    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="ca6ee48be7fe69b714035bacfe99898c38e6bceb824fe1c1fd65d7216be9dd8a" exitCode=0
Nov 02 12:26:42.507213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506949    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="d4c1ceb499f52323ec3e3c38ba0735486498a18b1c4b4c6631851b70c2db0fea" exitCode=0
Nov 02 12:26:42.507213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.506954    4928 generic.go:356] "Generic (PLEG): container finished" podID="2d0715df-bd6b-45ec-b51a-fb79504da0e5" containerID="ef704f8ff1a536c7f9677f1667bd0c4a367ad2963ac117900247835e6b3529cd" exitCode=0
Nov 02 12:26:42.509232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.509204    4928 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:42.511641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.511621    4928 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 12:26:42.511641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.511622    4928 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:42.525621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.525598    4928 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:42.525621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.525614    4928 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:42.540861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.540785    4928 generic.go:356] "Generic (PLEG): container finished" podID="10c19b4cc2c9735bf381d4249f2c0fe4" containerID="fdfc0866d0e449fe29b2d8e8ab01f8a6b9c674815d418c3ab12233f58701fa9a" exitCode=0
Nov 02 12:26:42.550281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.550265    4928 manager.go:324] Recovery completed
Nov 02 12:26:42.569924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.569904    4928 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:26:42.569924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.569919    4928 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:26:42.570066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.569936    4928 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:26:42.570097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570066    4928 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 12:26:42.570097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570077    4928 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 12:26:42.570097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570093    4928 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 12:26:42.570227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570099    4928 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 12:26:42.570227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570104    4928 policy_none.go:50] "Start"
Nov 02 12:26:42.570227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570112    4928 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:26:42.570227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570121    4928 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:26:42.570335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570248    4928 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 12:26:42.570335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570255    4928 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 12:26:42.570335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.570261    4928 policy_none.go:44] "Start"
Nov 02 12:26:42.574568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.574539    4928 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:26:42.577100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.577084    4928 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:26:42.577562 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.577110    4928 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:26:42.577612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.577565    4928 server.go:87] "Starting device plugin registration server"
Nov 02 12:26:42.577847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.577820    4928 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:26:42.577891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.577853    4928 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:26:42.577964 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.577946    4928 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:26:42.578077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.578050    4928 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:26:42.578157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.578085    4928 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:26:42.578157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.578090    4928 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:26:42.578252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.578145    4928 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:26:42.578074273Z","UUID":"06307d15-15e1-4e10-931a-0358e20763c0","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:26:42.579426 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.579405    4928 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:26:42.579918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.579901    4928 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:26:42.580010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.579931    4928 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:26:42.589255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.589216    4928 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"]
Nov 02 12:26:42.589948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.589904    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal" event={"ID":"fc7bed3dd43ef144d6694274c9c4b683","Type":"ContainerStarted","Data":"e1811119e0bc63cd2c425f3e15f81b05083582da776f6eec04102f2f90f16791"}
Nov 02 12:26:42.590030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.589948    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal" event={"ID":"fc7bed3dd43ef144d6694274c9c4b683","Type":"ContainerStarted","Data":"2bc71cbbcd1489d5728b674faf782187af5052459fe200ed0b99a09ef37cfa29"}
Nov 02 12:26:42.590030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.589984    4928 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.590119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.590056    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerStarted","Data":"5b3c68592a85762beb455156e5ce4584d9d62a3eec9084b4a110a37e2ef43c24"}
Nov 02 12:26:42.590119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.590076    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerDied","Data":"fdfc0866d0e449fe29b2d8e8ab01f8a6b9c674815d418c3ab12233f58701fa9a"}
Nov 02 12:26:42.590119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.590095    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal" event={"ID":"10c19b4cc2c9735bf381d4249f2c0fe4","Type":"ContainerStarted","Data":"b47870503b0a34f5a4d270d634f415f35f09c68e74e7cf3d7433872973f4c30f"}
Nov 02 12:26:42.590241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.590214    4928 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.615732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.615708    4928 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:42.615868 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.615757    4928 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.616315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.616301    4928 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:42.616354 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:42.616336    4928 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.678763 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.678732    4928 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:42.685978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.685955    4928 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:42.686094 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.685987    4928 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:42.686094 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.685999    4928 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-160.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:42.686094 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.686043    4928 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.713274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.713246    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.713274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.713275    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.713396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.713293    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.723028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.723007    4928 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.723093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.723083    4928 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813447    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813487    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813521    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813571    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc7bed3dd43ef144d6694274c9c4b683-config\") pod \"kube-apiserver-proxy-ip-10-0-135-160.ec2.internal\" (UID: \"fc7bed3dd43ef144d6694274c9c4b683\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813578    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:42.813767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:42.813574    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10c19b4cc2c9735bf381d4249f2c0fe4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" (UID: \"10c19b4cc2c9735bf381d4249f2c0fe4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:43.443070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.443033    4928 apiserver.go:52] "Watching apiserver"
Nov 02 12:26:43.488481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.488447    4928 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:26:43.489307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.489281    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7","kube-system/kas-connection-checker-685895c7c9-hzn62","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph","openshift-dns/dns-default-2928m","openshift-network-diagnostics/network-check-target-bx8dr","openshift-ingress/router-default-86d86d55f7-vpzjg","openshift-insights/insights-operator-7c4986d449-x6zzd","openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl","openshift-image-registry/image-registry-846c5f68d8-tjd87","openshift-image-registry/node-ca-cdn4d","openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4","openshift-insights/insights-runtime-extractor-qnhns","openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f","openshift-ingress-canary/ingress-canary-qsffd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal","kube-system/kas-connection-checker-685895c7c9-9nkss","openshift-console-operator/console-operator-7cd56b8dbc-55ghk","openshift-dns/node-resolver-2c8j4","kube-system/konnectivity-agent-dwphp","openshift-cluster-node-tuning-operator/tuned-rsbxh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh","openshift-multus/multus-22kwl","kube-system/kas-connection-checker-685895c7c9-m5bqj","openshift-multus/multus-additional-cni-plugins-mff82","openshift-network-console/networking-console-plugin-86c478bb88-8gt88","openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc","kube-system/global-pull-secret-syncer-qqgcj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm","openshift-ovn-kubernetes/ovnkube-node-97gdw","openshift-multus/network-metrics-daemon-d8xkp","kube-system/kube-apiserver-proxy-ip-10-0-135-160.ec2.internal","openshift-network-operator/iptables-alerter-hqnhw"]
Nov 02 12:26:43.490751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.490734    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.491082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.491060    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.491484 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.491470    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.495008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.494983    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.496850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.496817    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:43.498182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.498163    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:43.502003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.501985    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.502541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.502522    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.503724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.503708    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.513110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513087    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 12:26:43.513201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513150    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.513201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513158    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.513201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513165    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.513201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513183    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 12:26:43.513201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513195    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.513388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513206    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.513388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513220    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 12:26:43.513388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513184    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 12:26:43.513388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.513264    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 12:26:43.530962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.530935    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 12:26:43.531241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.531226    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 12:26:43.534895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.534881    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 12:26:43.537441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.537428    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.544368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.544352    4928 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:43.544654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.544634    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:26:43.549216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.549201    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.549335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.549324    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:26:43.551132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.551119    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:26:43.551261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.551249    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 12:26:43.551403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.551390    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:26:43.551652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.551641    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 12:26:43.551862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.551850    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:26:43.552250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.552239    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.556913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.556900    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 12:26:43.557001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.556982    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 12:26:43.557045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.556991    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.557144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.557128    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.557220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.557207    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.557220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.557217    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.559123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.559088    4928 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:26:43.567885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.567865    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.567885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.567866    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:26:43.568017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.567887    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.568156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.568137    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.568208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.568151    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.571660 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.571644    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.571801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.571707    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.571801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.571783    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wx6z7\""
Nov 02 12:26:43.572026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.572011    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.572079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.572034    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 12:26:43.572079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.572065    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.596520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596456    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 12:26:43.596520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596505    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.596949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596897    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.596949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596920    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 12:26:43.596949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596899    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.597135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596962    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.597135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.596993    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.597135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.597029    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:26:43.600263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600240    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.600263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600249    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.600555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600538    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:26:43.600555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600539    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:26:43.600696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600561    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.600859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600841    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:26:43.600951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600842    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:26:43.601013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600990    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:26:43.601013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.600993    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:26:43.601086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601016    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:26:43.601086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601078    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.601207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601194    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 12:26:43.601261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601220    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 12:26:43.601261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601226    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 12:26:43.601346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601277    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gp6gb\""
Nov 02 12:26:43.601346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601292    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.601346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601327    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 12:26:43.601432 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601363    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.601528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601506    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:26:43.601588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601533    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:26:43.601588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601535    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.601588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601547    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:26:43.601588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601579    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:26:43.601720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601669    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:26:43.601720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601717    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:26:43.601787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601721    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:26:43.601787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601747    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:26:43.601787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601757    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.601910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601786    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:26:43.601910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601786    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.601910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601839    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:26:43.601910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601852    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.601910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601867    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:26:43.602064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.601980    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.602064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.602016    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:26:43.602210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.602198    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:26:43.616192 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616175    4928 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:43.616375 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.616355    4928 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-160.ec2.internal"
Nov 02 12:26:43.616437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616379    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:26:43.616437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616400    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:26:43.616437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616410    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:26:43.616437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616403    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-htk9k\""
Nov 02 12:26:43.616437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616433    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:26:43.616696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616454    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:26:43.616696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616358    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:26:43.616696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616628    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.616696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616656    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.616696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616674    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8h2\" (UniqueName: \"kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616707    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616731    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616751    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616779    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616804    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616846    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.616961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616926    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.616994    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617007    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkmn\" (UniqueName: \"kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617026    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e586231-9b95-4114-82c2-602480b1e3c9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617040    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617056    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-serving-cert\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617072    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617121    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617150    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69559f09-3ce2-435a-a738-1d6892629b7d-konnectivity-ca\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617180    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617212    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617240    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617240    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.617292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617267    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617305    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617339    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6vh\" (UniqueName: \"kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617367    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7v44\" (UniqueName: \"kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617373    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617437    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9xd\" (UniqueName: \"kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617459    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-config\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617470    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6vg\" (UniqueName: \"kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617491    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617529    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617559    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617588    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkfd\" (UniqueName: \"kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617612    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617640    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617666    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617697    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617728    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617772    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qt9\" (UniqueName: \"kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9\") pod \"network-check-source-6f78d59f99-w9hwg\" (UID: \"f2849f75-3d38-49d2-8372-fd7776ddf12e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617802    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617841    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617848    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-env-overrides\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617868    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.617899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617893    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617916    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617941    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617940    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ceb68925-753a-4c72-8d6d-01b1e2550c6a-iptables-alerter-script\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617970    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ae75b6-203e-4d60-b728-412645e0a4a8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.617995    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a12625f-91bf-4fc3-a398-eaa065d811de-data-volume\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618020    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-script-lib\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618031    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618084    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618127    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618183    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618249    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618264    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68d7\" (UniqueName: \"kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618293    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618302    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-multus-daemon-config\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618316    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-service-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618321    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618354    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618379    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618420    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618442    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618461    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618493    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618536    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-tmp\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618575    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-tuned\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618574    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618582    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618598    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6da4be61-ac40-4531-9357-e05f84219250-serving-cert\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618612    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618644    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618670    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.618677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618691    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618705    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618726    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618754    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618791    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618809    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e2649ce-6913-40b6-a47a-cd71b5208dd3-serving-cert\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618837    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ppz\" (UniqueName: \"kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618870    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618955    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.618980    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619012    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619030    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619071    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619121    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619167    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619189    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-default-certificate\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619195    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619226    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619230    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d37dd0-8676-4b17-81a4-201865d84b89-serviceca\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619244    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovnkube-config\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619279    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvpt\" (UniqueName: \"kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt\") pod \"volume-data-source-validator-58c9f5f46d-xwbjh\" (UID: \"8250eede-bd12-46d3-a569-6a242bbfd566\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619365    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ae75b6-203e-4d60-b728-412645e0a4a8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619381    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619435    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619481    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619504    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnl7\" (UniqueName: \"kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619520    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619544    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619571    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbqt\" (UniqueName: \"kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt\") pod \"kas-connection-checker-685895c7c9-9nkss\" (UID: \"0e8b0b9d-80b8-460e-98d3-189c240cd20d\") " pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619610    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619644    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619668    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9939d8b4-1181-463b-a465-13da054067b6-tmp-dir\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619700    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619725    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619756    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76chf\" (UniqueName: \"kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf\") pod \"kas-connection-checker-685895c7c9-hzn62\" (UID: \"ce9a1148-d94a-47ad-8b2d-147af29a5797\") " pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619789    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.619817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619846    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvx4\" (UniqueName: \"kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4\") pod \"migrator-66df5d496b-vq98f\" (UID: \"e51414af-06a2-416d-9656-871c483ba22a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619878    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619901    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619918    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619948    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.619967    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620014    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620055    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620067    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620094    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620100    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620133    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblvc\" (UniqueName: \"kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620150    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620158    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620166    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620247    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620270    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620294    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-tmp\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620307    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs5x\" (UniqueName: \"kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620345    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620393    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620413    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620431    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620446    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620462    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620506    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620529    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620565    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620563    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086665ff-9fcc-48af-982e-59bb0239605b-cert\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620687    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620710    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-config\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620724    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wt\" (UniqueName: \"kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620751    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620777    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620806    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqrj\" (UniqueName: \"kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620856    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620935    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620995    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a76295dc-1849-4d96-8940-cdc1de45551e-cni-binary-copy\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.620990    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621031    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621044    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621159    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621206    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621226    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621254    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621270    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.621265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621287    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621319    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621329    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3c9141-625c-449d-8b94-3aa76c9a5d04-tmp-dir\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621347    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621374    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621401    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkwq\" (UniqueName: \"kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq\") pod \"kas-connection-checker-685895c7c9-m5bqj\" (UID: \"04856dfa-3c20-45b0-bc5f-f68896f87c51\") " pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621489    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621535    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3c9141-625c-449d-8b94-3aa76c9a5d04-config-volume\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621568    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr47\" (UniqueName: \"kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621594    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrgr\" (UniqueName: \"kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621623    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621649    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-stats-auth\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621649    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621684    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621701    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621729    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621758    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621778    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrnh\" (UniqueName: \"kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621799    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621838    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-ovn-node-metrics-cert\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621848    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vn2h\" (UniqueName: \"kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621895    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6da4be61-ac40-4531-9357-e05f84219250-snapshots\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621941    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.621952    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622031    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622062    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622113    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622148    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622166    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.622680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.622232    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69559f09-3ce2-435a-a738-1d6892629b7d-agent-certs\") pod \"konnectivity-agent-dwphp\" (UID: \"69559f09-3ce2-435a-a738-1d6892629b7d\") " pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:26:43.627137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627121    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:26:43.627179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627148    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnbq6\""
Nov 02 12:26:43.627813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627800    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r2cgm\""
Nov 02 12:26:43.627894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627818    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 12:26:43.627894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627838    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 12:26:43.627894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627855    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 12:26:43.627990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.627941    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:26:43.628099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.628082    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cf5eb1e-e80f-40d5-9367-4af0c929723f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.629126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.629108    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.629428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.629413    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:26:43.630559 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.630543    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e2649ce-6913-40b6-a47a-cd71b5208dd3-trusted-ca\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.632111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.632094    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.632296 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.632282    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b28cf8-a271-41c8-a765-811b45bfeeb0-original-pull-secret\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.641640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.638495    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6da4be61-ac40-4531-9357-e05f84219250-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.649719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.649699    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkfd\" (UniqueName: \"kubernetes.io/projected/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-api-access-smkfd\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.649719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.649709    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblvc\" (UniqueName: \"kubernetes.io/projected/a76295dc-1849-4d96-8940-cdc1de45551e-kube-api-access-gblvc\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.649949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.649933    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6vh\" (UniqueName: \"kubernetes.io/projected/ceb68925-753a-4c72-8d6d-01b1e2550c6a-kube-api-access-9x6vh\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.656006 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.655987    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9xd\" (UniqueName: \"kubernetes.io/projected/0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151-kube-api-access-dw9xd\") pod \"service-ca-operator-544d749c8c-gj2h4\" (UID: \"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4"
Nov 02 12:26:43.658183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.658164    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6vg\" (UniqueName: \"kubernetes.io/projected/82d00d41-1c4a-450f-9a27-8571dc21bf21-kube-api-access-hh6vg\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.658375 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.658357    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr47\" (UniqueName: \"kubernetes.io/projected/6da4be61-ac40-4531-9357-e05f84219250-kube-api-access-bnr47\") pod \"insights-operator-7c4986d449-x6zzd\" (UID: \"6da4be61-ac40-4531-9357-e05f84219250\") " pod="openshift-insights/insights-operator-7c4986d449-x6zzd"
Nov 02 12:26:43.668119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.667987    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrgr\" (UniqueName: \"kubernetes.io/projected/66807b2c-d92d-48aa-815d-4b437efae8eb-kube-api-access-pxrgr\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:43.668119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668069    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvx4\" (UniqueName: \"kubernetes.io/projected/e51414af-06a2-416d-9656-871c483ba22a-kube-api-access-5qvx4\") pod \"migrator-66df5d496b-vq98f\" (UID: \"e51414af-06a2-416d-9656-871c483ba22a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vq98f"
Nov 02 12:26:43.668119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668080    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqrj\" (UniqueName: \"kubernetes.io/projected/10ae75b6-203e-4d60-b728-412645e0a4a8-kube-api-access-prqrj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl\" (UID: \"10ae75b6-203e-4d60-b728-412645e0a4a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl"
Nov 02 12:26:43.668443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668294    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.668443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668309    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkmn\" (UniqueName: \"kubernetes.io/projected/c6ee3214-5327-4181-8c06-5a9c350ed235-kube-api-access-4zkmn\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.668550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668488    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68d7\" (UniqueName: \"kubernetes.io/projected/086665ff-9fcc-48af-982e-59bb0239605b-kube-api-access-m68d7\") pod \"ingress-canary-qsffd\" (UID: \"086665ff-9fcc-48af-982e-59bb0239605b\") " pod="openshift-ingress-canary/ingress-canary-qsffd"
Nov 02 12:26:43.668550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668542    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8h2\" (UniqueName: \"kubernetes.io/projected/9939d8b4-1181-463b-a465-13da054067b6-kube-api-access-jc8h2\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.668988 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.668970    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7v44\" (UniqueName: \"kubernetes.io/projected/3292ca18-a18b-404e-bf25-e2d7cfb9d383-kube-api-access-n7v44\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.669210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.669194    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7cf5eb1e-e80f-40d5-9367-4af0c929723f-kube-api-access-hhwpv\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.670236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.670221    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.671003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.670987    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qt9\" (UniqueName: \"kubernetes.io/projected/f2849f75-3d38-49d2-8372-fd7776ddf12e-kube-api-access-s2qt9\") pod \"network-check-source-6f78d59f99-w9hwg\" (UID: \"f2849f75-3d38-49d2-8372-fd7776ddf12e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-w9hwg"
Nov 02 12:26:43.673326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.673309    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.675481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.675462    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vn2h\" (UniqueName: \"kubernetes.io/projected/a7d37dd0-8676-4b17-81a4-201865d84b89-kube-api-access-7vn2h\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.675550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.675479    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7nf\" (UniqueName: \"kubernetes.io/projected/45c4cc29-c98d-4fd9-b6d8-8090d72a11ab-kube-api-access-mn7nf\") pod \"network-check-target-bx8dr\" (UID: \"45c4cc29-c98d-4fd9-b6d8-8090d72a11ab\") " pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:43.675605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.675564    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs5x\" (UniqueName: \"kubernetes.io/projected/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kube-api-access-bqs5x\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.676074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.676055    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvpt\" (UniqueName: \"kubernetes.io/projected/8250eede-bd12-46d3-a569-6a242bbfd566-kube-api-access-jcvpt\") pod \"volume-data-source-validator-58c9f5f46d-xwbjh\" (UID: \"8250eede-bd12-46d3-a569-6a242bbfd566\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xwbjh"
Nov 02 12:26:43.677388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.677372    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wt\" (UniqueName: \"kubernetes.io/projected/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-kube-api-access-rk6wt\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.678459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.678438    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbqt\" (UniqueName: \"kubernetes.io/projected/0e8b0b9d-80b8-460e-98d3-189c240cd20d-kube-api-access-cwbqt\") pod \"kas-connection-checker-685895c7c9-9nkss\" (UID: \"0e8b0b9d-80b8-460e-98d3-189c240cd20d\") " pod="kube-system/kas-connection-checker-685895c7c9-9nkss"
Nov 02 12:26:43.678520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.678470    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnl7\" (UniqueName: \"kubernetes.io/projected/7e3c9141-625c-449d-8b94-3aa76c9a5d04-kube-api-access-zbnl7\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.678961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.678943    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrnh\" (UniqueName: \"kubernetes.io/projected/8e2649ce-6913-40b6-a47a-cd71b5208dd3-kube-api-access-qqrnh\") pod \"console-operator-7cd56b8dbc-55ghk\" (UID: \"8e2649ce-6913-40b6-a47a-cd71b5208dd3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:43.679044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.679018    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76chf\" (UniqueName: \"kubernetes.io/projected/ce9a1148-d94a-47ad-8b2d-147af29a5797-kube-api-access-76chf\") pod \"kas-connection-checker-685895c7c9-hzn62\" (UID: \"ce9a1148-d94a-47ad-8b2d-147af29a5797\") " pod="kube-system/kas-connection-checker-685895c7c9-hzn62"
Nov 02 12:26:43.679472 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.679457    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ppz\" (UniqueName: \"kubernetes.io/projected/2d0715df-bd6b-45ec-b51a-fb79504da0e5-kube-api-access-m6ppz\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.680083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.680069    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.680681 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.680661    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkwq\" (UniqueName: \"kubernetes.io/projected/04856dfa-3c20-45b0-bc5f-f68896f87c51-kube-api-access-bpkwq\") pod \"kas-connection-checker-685895c7c9-m5bqj\" (UID: \"04856dfa-3c20-45b0-bc5f-f68896f87c51\") " pod="kube-system/kas-connection-checker-685895c7c9-m5bqj"
Nov 02 12:26:43.722446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722424    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.722551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722450    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:43.722551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722465    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.722551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722479    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.722551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722494    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:43.722551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722543    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-etc-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722554    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722573    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722583    4928 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722614    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722642    4928 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722643    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722660    4928 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722673    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.222638007 +0000 UTC m=+1.871691967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722675    4928 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722710    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.222690903 +0000 UTC m=+1.871744854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : secret "metrics-daemon-secret" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722713    4928 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722730    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722708    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-multus\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722762    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.222747345 +0000 UTC m=+1.871801311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722780    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.222770882 +0000 UTC m=+1.871824819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722795    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722805    4928 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:43.722807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722815    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722856    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722885    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722899    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.222882729 +0000 UTC m=+1.871936665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.722982    4928 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723003    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722910    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a12625f-91bf-4fc3-a398-eaa065d811de-crio-socket\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722928    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysconfig\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.723039    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.223025141 +0000 UTC m=+1.872079102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723030    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723066    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-log-socket\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723071    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723076    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723105    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a12625f-91bf-4fc3-a398-eaa065d811de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723122    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723132    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-dbus\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.722951    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-device-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723152    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723153    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-run\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723196    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-netd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723225    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723249    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723260    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-var-lib-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723267    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723290    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-etc-kubernetes\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723313    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-registration-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723317    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723345    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723368    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723391    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723396    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-k8s-cni-cncf-io\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723416    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723441    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-ovn\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723448    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723444    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723467    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-conf-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723479    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-kubelet\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723487    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723496    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-modprobe-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723511    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723555    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723558    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-cni-bin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723579    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723582    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-sys-fs\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723601    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723603    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-system-cni-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723618    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-sys\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.723605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723635    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723659    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723685    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723711    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-node-log\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723710    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723737    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723752    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723759    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d37dd0-8676-4b17-81a4-201865d84b89-host\") pod \"node-ca-cdn4d\" (UID: \"a7d37dd0-8676-4b17-81a4-201865d84b89\") " pod="openshift-image-registry/node-ca-cdn4d"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723779    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723783    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-cnibin\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723639    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-cni-bin\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723806    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723814    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-systemd\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723851    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723878    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723902    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723910    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-d\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723925    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723964    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723988    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-sysctl-conf\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.723991    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724017    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724042    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724050    4928 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724065    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724089    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.224079896 +0000 UTC m=+1.873133832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724101    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724118    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724137    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724138    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-socket-dir\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724152    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724169    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724187    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724204    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724220    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724237    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724254    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724270    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724284    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724301    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724333    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724324    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-var-lib-kubelet\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724353    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724371    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724385    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-slash\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724435    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-systemd\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724440    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-multus-socket-dir-parent\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724470    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-cnibin\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724478    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-run-openvswitch\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724499    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-etc-kubernetes\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724530    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9939d8b4-1181-463b-a465-13da054067b6-hosts-file\") pod \"node-resolver-2c8j4\" (UID: \"9939d8b4-1181-463b-a465-13da054067b6\") " pod="openshift-dns/node-resolver-2c8j4"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724542    4928 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724567    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.224559068 +0000 UTC m=+1.873613003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724573    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-systemd-units\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724587    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-multus-certs\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724187    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee0b7f5d-71cf-4491-a17b-63cb47bb6441-host-run-netns\") pod \"ovnkube-node-97gdw\" (UID: \"ee0b7f5d-71cf-4491-a17b-63cb47bb6441\") " pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724611    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-var-lib-kubelet\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724626    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b28cf8-a271-41c8-a765-811b45bfeeb0-kubelet-config\") pod \"global-pull-secret-syncer-qqgcj\" (UID: \"c5b28cf8-a271-41c8-a765-811b45bfeeb0\") " pod="kube-system/global-pull-secret-syncer-qqgcj"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724283    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-lib-modules\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724652    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb68925-753a-4c72-8d6d-01b1e2550c6a-host-slash\") pod \"iptables-alerter-hqnhw\" (UID: \"ceb68925-753a-4c72-8d6d-01b1e2550c6a\") " pod="openshift-network-operator/iptables-alerter-hqnhw"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724661    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e8a733-2eca-4192-a6ab-67ccc40ff2ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-npdph\" (UID: \"d7e8a733-2eca-4192-a6ab-67ccc40ff2ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npdph"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724676    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-hostroot\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724723    4928 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724730    4928 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724755    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.224741182 +0000 UTC m=+1.873795139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724761    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-host-run-netns\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:43.724773    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:44.224764089 +0000 UTC m=+1.873818025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724791    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0715df-bd6b-45ec-b51a-fb79504da0e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mff82\" (UID: \"2d0715df-bd6b-45ec-b51a-fb79504da0e5\") " pod="openshift-multus/multus-additional-cni-plugins-mff82"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724811    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ee3214-5327-4181-8c06-5a9c350ed235-host\") pod \"tuned-rsbxh\" (UID: \"c6ee3214-5327-4181-8c06-5a9c350ed235\") " pod="openshift-cluster-node-tuning-operator/tuned-rsbxh"
Nov 02 12:26:43.725397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.724864    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a76295dc-1849-4d96-8940-cdc1de45551e-system-cni-dir\") pod \"multus-22kwl\" (UID: \"a76295dc-1849-4d96-8940-cdc1de45551e\") " pod="openshift-multus/multus-22kwl"
Nov 02 12:26:43.792615 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:43.792581    4928 scope.go:122] "RemoveContainer" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"
Nov 02 12:26:44.229093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229003    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:44.229093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:44.229093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229069    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229133    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229147    4928 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229158    4928 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78fdc47b4f-dp8t7: secret "image-registry-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229183    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229196    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls podName:532cb38b-3ea7-433c-88de-fb41dba50795 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229185926 +0000 UTC m=+2.878239876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls") pod "image-registry-78fdc47b4f-dp8t7" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795") : secret "image-registry-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229215    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229209956 +0000 UTC m=+2.878263893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : configmap references non-existent config key: service-ca.crt
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229228    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229253    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229256    4928 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229278    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229307    4928 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229308    4928 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229319    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229304296 +0000 UTC m=+2.878358264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229308    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229259    4928 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229347    4928 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846c5f68d8-tjd87: secret "image-registry-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229266    4928 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:44.229360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229348    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs podName:66807b2c-d92d-48aa-815d-4b437efae8eb nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229335721 +0000 UTC m=+2.878389670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs") pod "network-metrics-daemon-d8xkp" (UID: "66807b2c-d92d-48aa-815d-4b437efae8eb") : secret "metrics-daemon-secret" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229395    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229382742 +0000 UTC m=+2.878436679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229396    4928 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229420    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls podName:3001af94-ab6c-4995-b400-edecd5c02f59 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229409039 +0000 UTC m=+2.878462978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls") pod "image-registry-846c5f68d8-tjd87" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59") : secret "image-registry-tls" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229400    4928 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229437    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls podName:7e3c9141-625c-449d-8b94-3aa76c9a5d04 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229429961 +0000 UTC m=+2.878483897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls") pod "dns-default-2928m" (UID: "7e3c9141-625c-449d-8b94-3aa76c9a5d04") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229457    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs podName:82d00d41-1c4a-450f-9a27-8571dc21bf21 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.22944692 +0000 UTC m=+2.878500856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs") pod "router-default-86d86d55f7-vpzjg" (UID: "82d00d41-1c4a-450f-9a27-8571dc21bf21") : secret "router-metrics-certs-default" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229476    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229468467 +0000 UTC m=+2.878522409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.229563    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229635    4928 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:44.230056 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.229663    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.229655954 +0000 UTC m=+2.878709926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:44.325890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.325864    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2c8j4_9939d8b4-1181-463b-a465-13da054067b6/dns-node-resolver/0.log"
Nov 02 12:26:44.551241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.551212    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/2.log"
Nov 02 12:26:44.551637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.551570    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/1.log"
Nov 02 12:26:44.551637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.551599    4928 generic.go:356] "Generic (PLEG): container finished" podID="8e2649ce-6913-40b6-a47a-cd71b5208dd3" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5" exitCode=255
Nov 02 12:26:44.551703 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.551675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerDied","Data":"18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"}
Nov 02 12:26:44.551741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.551727    4928 scope.go:122] "RemoveContainer" containerID="b125e50a166b0f3b89b3760624c5358b571578093bc5d8036c3799542d56d12b"
Nov 02 12:26:44.552108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:44.552083    4928 scope.go:122] "RemoveContainer" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"
Nov 02 12:26:44.552276 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:44.552259    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:45.238781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.238744    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:45.238962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.238796    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:45.238962 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.238932    4928 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:45.239065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.238937    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:45.239065 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239005    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.238988109 +0000 UTC m=+4.888042070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:45.239065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:45.239187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239090    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:45.239187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239120    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:45.239187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239148    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:45.239324 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239258    4928 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:45.239373 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239329    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.239309888 +0000 UTC m=+4.888363841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:45.239373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239326    4928 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:26:45.239373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239355    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:45.239546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239390    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:45.239546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239417    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:45.239546 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239516    4928 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.239683 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239559    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.239545082 +0000 UTC m=+4.888599032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.239683 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239565    4928 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:45.239683 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:45.239629    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls podName:7cf5eb1e-e80f-40d5-9367-4af0c929723f nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.239610659 +0000 UTC m=+4.888664613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d48zc" (UID: "7cf5eb1e-e80f-40d5-9367-4af0c929723f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 12:26:45.239855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.239688    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82d00d41-1c4a-450f-9a27-8571dc21bf21-service-ca-bundle\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:45.241538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.241507    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66807b2c-d92d-48aa-815d-4b437efae8eb-metrics-certs\") pod \"network-metrics-daemon-d8xkp\" (UID: \"66807b2c-d92d-48aa-815d-4b437efae8eb\") " pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:45.241670 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.241620    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3c9141-625c-449d-8b94-3aa76c9a5d04-metrics-tls\") pod \"dns-default-2928m\" (UID: \"7e3c9141-625c-449d-8b94-3aa76c9a5d04\") " pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:45.241817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.241795    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"image-registry-78fdc47b4f-dp8t7\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") " pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:45.242046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.242027    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"image-registry-846c5f68d8-tjd87\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") " pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:45.242100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.242049    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82d00d41-1c4a-450f-9a27-8571dc21bf21-metrics-certs\") pod \"router-default-86d86d55f7-vpzjg\" (UID: \"82d00d41-1c4a-450f-9a27-8571dc21bf21\") " pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:45.292120 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.292094    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:45.295527 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.295509    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:45.298888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.298872    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8xkp"
Nov 02 12:26:45.303264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.303237    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:45.303359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.303345    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:45.524050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.523979    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdn4d_a7d37dd0-8676-4b17-81a4-201865d84b89/node-ca/0.log"
Nov 02 12:26:45.556524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.556498    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/2.log"
Nov 02 12:26:45.621543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.621192    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86d86d55f7-vpzjg"]
Nov 02 12:26:45.624506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.624485    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8xkp"]
Nov 02 12:26:45.626347 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:45.626316    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d00d41_1c4a_450f_9a27_8571dc21bf21.slice/crio-f1c025e06d160e8100b8fb0264a3ef1a9bb6ce18a3a23cf6c1c82cd611a909ed WatchSource:0}: Error finding container f1c025e06d160e8100b8fb0264a3ef1a9bb6ce18a3a23cf6c1c82cd611a909ed: Status 404 returned error can't find the container with id f1c025e06d160e8100b8fb0264a3ef1a9bb6ce18a3a23cf6c1c82cd611a909ed
Nov 02 12:26:45.627500 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:45.627479    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66807b2c_d92d_48aa_815d_4b437efae8eb.slice/crio-4f39a93409af0c025304256a76c9c4b82adc3b5d8115d68e03bd3928cdfce1e7 WatchSource:0}: Error finding container 4f39a93409af0c025304256a76c9c4b82adc3b5d8115d68e03bd3928cdfce1e7: Status 404 returned error can't find the container with id 4f39a93409af0c025304256a76c9c4b82adc3b5d8115d68e03bd3928cdfce1e7
Nov 02 12:26:45.629485 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.629467    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:26:45.636123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.635999    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2928m"]
Nov 02 12:26:45.639574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.639538    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:26:45.640368 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:45.640342    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3c9141_625c_449d_8b94_3aa76c9a5d04.slice/crio-47e999cf9dbb9d03f3d530688de0372855fce944372660a4ff5a0fcfa250efc5 WatchSource:0}: Error finding container 47e999cf9dbb9d03f3d530688de0372855fce944372660a4ff5a0fcfa250efc5: Status 404 returned error can't find the container with id 47e999cf9dbb9d03f3d530688de0372855fce944372660a4ff5a0fcfa250efc5
Nov 02 12:26:45.643460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:45.643437    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:26:45.654913 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:45.654891    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532cb38b_3ea7_433c_88de_fb41dba50795.slice/crio-3c627f5d1f4757ef2bac846cb516f0e41e9f92ef38d8ed737f3155a23c5348f4 WatchSource:0}: Error finding container 3c627f5d1f4757ef2bac846cb516f0e41e9f92ef38d8ed737f3155a23c5348f4: Status 404 returned error can't find the container with id 3c627f5d1f4757ef2bac846cb516f0e41e9f92ef38d8ed737f3155a23c5348f4
Nov 02 12:26:46.078396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.078358    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:26:46.088084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.088056    4928 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:46.088392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.088377    4928 scope.go:122] "RemoveContainer" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"
Nov 02 12:26:46.088584 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:46.088563    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:46.325419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.325369    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qsffd_086665ff-9fcc-48af-982e-59bb0239605b/serve-healthcheck-canary/0.log"
Nov 02 12:26:46.562759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.562718    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8xkp" event={"ID":"66807b2c-d92d-48aa-815d-4b437efae8eb","Type":"ContainerStarted","Data":"4f39a93409af0c025304256a76c9c4b82adc3b5d8115d68e03bd3928cdfce1e7"}
Nov 02 12:26:46.564241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.564210    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2928m" event={"ID":"7e3c9141-625c-449d-8b94-3aa76c9a5d04","Type":"ContainerStarted","Data":"47e999cf9dbb9d03f3d530688de0372855fce944372660a4ff5a0fcfa250efc5"}
Nov 02 12:26:46.565964 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.565931    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" event={"ID":"532cb38b-3ea7-433c-88de-fb41dba50795","Type":"ContainerStarted","Data":"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"}
Nov 02 12:26:46.566097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.565990    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" event={"ID":"532cb38b-3ea7-433c-88de-fb41dba50795","Type":"ContainerStarted","Data":"3c627f5d1f4757ef2bac846cb516f0e41e9f92ef38d8ed737f3155a23c5348f4"}
Nov 02 12:26:46.568232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.568165    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86d86d55f7-vpzjg" event={"ID":"82d00d41-1c4a-450f-9a27-8571dc21bf21","Type":"ContainerStarted","Data":"0b0c86d73097d18d9481dc350d383aea68aa42ad8dd1410e89816d3686a95d47"}
Nov 02 12:26:46.568232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.568207    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86d86d55f7-vpzjg" event={"ID":"82d00d41-1c4a-450f-9a27-8571dc21bf21","Type":"ContainerStarted","Data":"f1c025e06d160e8100b8fb0264a3ef1a9bb6ce18a3a23cf6c1c82cd611a909ed"}
Nov 02 12:26:46.570046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.570019    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87" event={"ID":"3001af94-ab6c-4995-b400-edecd5c02f59","Type":"ContainerStarted","Data":"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"}
Nov 02 12:26:46.570163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.570067    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87" event={"ID":"3001af94-ab6c-4995-b400-edecd5c02f59","Type":"ContainerStarted","Data":"096082a01425ca7052f20f695fd293158cf8a3c8b761d8cbc277b999fd3378d8"}
Nov 02 12:26:46.729804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.728906    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vq98f_e51414af-06a2-416d-9656-871c483ba22a/migrator/0.log"
Nov 02 12:26:46.929304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.929232    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl_10ae75b6-203e-4d60-b728-412645e0a4a8/kube-storage-version-migrator-operator/0.log"
Nov 02 12:26:46.989674 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.989643    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:46.990096 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:46.990077    4928 scope.go:122] "RemoveContainer" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"
Nov 02 12:26:46.990312 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:46.990292    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:47.258964 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.258933    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:47.259064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.258982    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:47.259064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.259021    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:47.259163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.259095    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:47.259284 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.259260    4928 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:47.259409 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.259331    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:51.259311612 +0000 UTC m=+8.908365568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:47.259747 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.259723    4928 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 12:26:47.259789 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.259782    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls podName:3292ca18-a18b-404e-bf25-e2d7cfb9d383 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:51.259766156 +0000 UTC m=+8.908820109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tk4bm" (UID: "3292ca18-a18b-404e-bf25-e2d7cfb9d383") : secret "samples-operator-tls" not found
Nov 02 12:26:47.260259 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.260237    4928 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:47.260341 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:47.260300    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:51.260283989 +0000 UTC m=+8.909337926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:47.262459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.262435    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cf5eb1e-e80f-40d5-9367-4af0c929723f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d48zc\" (UID: \"7cf5eb1e-e80f-40d5-9367-4af0c929723f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:47.292539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.292505    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:47.295518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.295499    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:47.392903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.392876    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"
Nov 02 12:26:47.517510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.517480    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:47.518775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.518752    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86d86d55f7-vpzjg"
Nov 02 12:26:47.576138 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.576103    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8xkp" event={"ID":"66807b2c-d92d-48aa-815d-4b437efae8eb","Type":"ContainerStarted","Data":"bd094bc938c1c78948c79feda0c2ccf8588745152625d8ad6d3273b6f11ce6a8"}
Nov 02 12:26:47.576574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.576153    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8xkp" event={"ID":"66807b2c-d92d-48aa-815d-4b437efae8eb","Type":"ContainerStarted","Data":"d34cf6c040315dd3b318e4a78540cca3ae984aa03af8daba20c4dca5d6c55795"}
Nov 02 12:26:47.578150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.578128    4928 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:47.578150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.578146    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2928m" event={"ID":"7e3c9141-625c-449d-8b94-3aa76c9a5d04","Type":"ContainerStarted","Data":"3883d2197436cefe05be8f01e5c012cbe26736863875ef95afb20cc9764cd7dc"}
Nov 02 12:26:47.578306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.578203    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2928m" event={"ID":"7e3c9141-625c-449d-8b94-3aa76c9a5d04","Type":"ContainerStarted","Data":"c11f2eccfe6322f98e9c0291628e1359f861c9f307d0643c51df17b6d032c311"}
Nov 02 12:26:47.578306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.578157    4928 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:47.707270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:47.707235    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc"]
Nov 02 12:26:47.711044 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:47.711016    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf5eb1e_e80f_40d5_9367_4af0c929723f.slice/crio-7083c53632e2fb00029f8316f24be59687045c51fb16541130f3f3b8e05aec06 WatchSource:0}: Error finding container 7083c53632e2fb00029f8316f24be59687045c51fb16541130f3f3b8e05aec06: Status 404 returned error can't find the container with id 7083c53632e2fb00029f8316f24be59687045c51fb16541130f3f3b8e05aec06
Nov 02 12:26:48.063219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:48.063184    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:48.583574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:48.583529    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc" event={"ID":"7cf5eb1e-e80f-40d5-9367-4af0c929723f","Type":"ContainerStarted","Data":"7083c53632e2fb00029f8316f24be59687045c51fb16541130f3f3b8e05aec06"}
Nov 02 12:26:48.594398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:48.594375    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:49.588525 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:49.588466    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d48zc" event={"ID":"7cf5eb1e-e80f-40d5-9367-4af0c929723f","Type":"ContainerStarted","Data":"b6a21af889a12eba556df75a708559a97c56cbaa6c76f4043684298164379bff"}
Nov 02 12:26:50.712631 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:50.712594    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:26:50.713012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:50.712794    4928 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:51.222575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.222540    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:26:51.295225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.295194    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:51.295360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.295246    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:51.295360 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:51.295345    4928 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 12:26:51.295449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.295360    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:51.295449 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:51.295404    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert podName:5e586231-9b95-4114-82c2-602480b1e3c9 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:59.295389943 +0000 UTC m=+16.944443879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8gt88" (UID: "5e586231-9b95-4114-82c2-602480b1e3c9") : secret "networking-console-plugin-cert" not found
Nov 02 12:26:51.295517 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:51.295467    4928 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:51.295546 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:51.295528    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls podName:9a12625f-91bf-4fc3-a398-eaa065d811de nodeName:}" failed. No retries permitted until 2026-11-02 12:26:59.295513195 +0000 UTC m=+16.944567133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnhns" (UID: "9a12625f-91bf-4fc3-a398-eaa065d811de") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:51.297640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.297623    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3292ca18-a18b-404e-bf25-e2d7cfb9d383-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tk4bm\" (UID: \"3292ca18-a18b-404e-bf25-e2d7cfb9d383\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:51.591451 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.591403    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"
Nov 02 12:26:51.888425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:51.888349    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm"]
Nov 02 12:26:52.599652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:52.599613    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm" event={"ID":"3292ca18-a18b-404e-bf25-e2d7cfb9d383","Type":"ContainerStarted","Data":"440a834f2634fb3e9fb17d622279817ebb9f5e86a577d41654438cd5fcb5bb38"}
Nov 02 12:26:52.833661 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:52.833628    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:26:53.604936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:53.604901    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm" event={"ID":"3292ca18-a18b-404e-bf25-e2d7cfb9d383","Type":"ContainerStarted","Data":"ea9d1a242619023101202d8349ae556dbddb82ce30592db6ce2931ddae7b3f98"}
Nov 02 12:26:54.609761 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:54.609724    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tk4bm" event={"ID":"3292ca18-a18b-404e-bf25-e2d7cfb9d383","Type":"ContainerStarted","Data":"d6cfd660ebcc186c7218b863b2d6795f690c10fe53bcbb1e890bd69fa1ff1733"}
Nov 02 12:26:55.490041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.490009    4928 scope.go:122] "RemoveContainer" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"
Nov 02 12:26:55.614412 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.614384    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:26:55.614876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.614858    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/2.log"
Nov 02 12:26:55.614935 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.614893    4928 generic.go:356] "Generic (PLEG): container finished" podID="8e2649ce-6913-40b6-a47a-cd71b5208dd3" containerID="670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1" exitCode=255
Nov 02 12:26:55.614988 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.614967    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerDied","Data":"670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1"}
Nov 02 12:26:55.615485 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.615465    4928 scope.go:122] "RemoveContainer" containerID="18801d40ee1a7f34c37220e1793be82c9ffc91d58aef79202b2e7c6da1163dc5"
Nov 02 12:26:55.615575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:55.615559    4928 scope.go:122] "RemoveContainer" containerID="670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1"
Nov 02 12:26:55.615796 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:55.615770    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:56.088491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:56.088454    4928 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:56.620397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:56.620364    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:26:56.620917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:56.620687    4928 scope.go:122] "RemoveContainer" containerID="670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1"
Nov 02 12:26:56.620917 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:56.620884    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:56.989896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:56.989868    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:26:57.426910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:57.426884    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:26:57.624486 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:57.624454    4928 scope.go:122] "RemoveContainer" containerID="670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1"
Nov 02 12:26:57.624915 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:26:57.624638    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-55ghk_openshift-console-operator(8e2649ce-6913-40b6-a47a-cd71b5208dd3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" podUID="8e2649ce-6913-40b6-a47a-cd71b5208dd3"
Nov 02 12:26:58.591034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:58.591002    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2928m"
Nov 02 12:26:59.370274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.370230    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:59.370655 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.370317    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:59.372648 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.372619    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a12625f-91bf-4fc3-a398-eaa065d811de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnhns\" (UID: \"9a12625f-91bf-4fc3-a398-eaa065d811de\") " pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:59.372757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.372710    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e586231-9b95-4114-82c2-602480b1e3c9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8gt88\" (UID: \"5e586231-9b95-4114-82c2-602480b1e3c9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:59.397608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.397585    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88"
Nov 02 12:26:59.404262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.404245    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnhns"
Nov 02 12:26:59.696788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.696706    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8gt88"]
Nov 02 12:26:59.704776 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:59.704746    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e586231_9b95_4114_82c2_602480b1e3c9.slice/crio-386fffd811ac73e664a6f5848473460a6c452f851029ddbfcea7cf6fc7d25c2e WatchSource:0}: Error finding container 386fffd811ac73e664a6f5848473460a6c452f851029ddbfcea7cf6fc7d25c2e: Status 404 returned error can't find the container with id 386fffd811ac73e664a6f5848473460a6c452f851029ddbfcea7cf6fc7d25c2e
Nov 02 12:26:59.711854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:26:59.711814    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnhns"]
Nov 02 12:26:59.714847 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:26:59.714808    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a12625f_91bf_4fc3_a398_eaa065d811de.slice/crio-3831f9760e1f9948e59907b830f0f3142a759e17ea69a8b3b89a7643f4cab408 WatchSource:0}: Error finding container 3831f9760e1f9948e59907b830f0f3142a759e17ea69a8b3b89a7643f4cab408: Status 404 returned error can't find the container with id 3831f9760e1f9948e59907b830f0f3142a759e17ea69a8b3b89a7643f4cab408
Nov 02 12:27:00.204024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:00.203989    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwphp"
Nov 02 12:27:00.637863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:00.637809    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnhns" event={"ID":"9a12625f-91bf-4fc3-a398-eaa065d811de","Type":"ContainerStarted","Data":"4f62d60f3596e49576b872d70ed1ba65bf0527879edf28f557c3c36301f4a369"}
Nov 02 12:27:00.638251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:00.637895    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnhns" event={"ID":"9a12625f-91bf-4fc3-a398-eaa065d811de","Type":"ContainerStarted","Data":"dadc4d9e1f8f6d74cb080781c5302c86953007338e29f81676ac677faee10e3a"}
Nov 02 12:27:00.638251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:00.637920    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnhns" event={"ID":"9a12625f-91bf-4fc3-a398-eaa065d811de","Type":"ContainerStarted","Data":"3831f9760e1f9948e59907b830f0f3142a759e17ea69a8b3b89a7643f4cab408"}
Nov 02 12:27:00.639065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:00.639034    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88" event={"ID":"5e586231-9b95-4114-82c2-602480b1e3c9","Type":"ContainerStarted","Data":"386fffd811ac73e664a6f5848473460a6c452f851029ddbfcea7cf6fc7d25c2e"}
Nov 02 12:27:01.227519 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:01.227475    4928 patch_prober.go:29] interesting pod/image-registry-78fdc47b4f-dp8t7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 12:27:01.227714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:01.227550    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:27:01.646320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:01.646274    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8gt88" event={"ID":"5e586231-9b95-4114-82c2-602480b1e3c9","Type":"ContainerStarted","Data":"aa684b1c5c00599dca39674f9f71c3d9c34d3da67cdd091a19d345230fe566a6"}
Nov 02 12:27:02.651238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:02.651198    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnhns" event={"ID":"9a12625f-91bf-4fc3-a398-eaa065d811de","Type":"ContainerStarted","Data":"0da21b22218049bf516ec5c0f01d36bedc37b17045d3f142a10b408672f0cbc1"}
Nov 02 12:27:04.053184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:04.053149    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97gdw"
Nov 02 12:27:04.570613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:04.570581    4928 patch_prober.go:29] interesting pod/image-registry-78fdc47b4f-dp8t7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 12:27:04.570775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:04.570631    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:27:05.066584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.066532    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jwcrl"]
Nov 02 12:27:05.071032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.071009    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.092064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.092035    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 12:27:05.092177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.092082    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89s57\""
Nov 02 12:27:05.101009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.100988    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 12:27:05.120032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120002    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-metrics-client-ca\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120040    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-root\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120115    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-textfile\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120224    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-wtmp\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120264    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-sys\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120293    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120356    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rlwq\" (UniqueName: \"kubernetes.io/projected/8e1addee-ee5f-456d-be31-59caef398724-kube-api-access-9rlwq\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120421    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.120493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.120482    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-tls\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.156400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.156372    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 12:27:05.156584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.156448    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 12:27:05.221943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.221909    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.221959    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-tls\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222000    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-metrics-client-ca\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222018    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-root\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222035    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-textfile\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222059    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-wtmp\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222075    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-sys\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222096    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222128    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rlwq\" (UniqueName: \"kubernetes.io/projected/8e1addee-ee5f-456d-be31-59caef398724-kube-api-access-9rlwq\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222141    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-root\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222171    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-sys\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222323    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-wtmp\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222503    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-textfile\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222679    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-metrics-client-ca\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.222741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.222702    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.224461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.224444    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-tls\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.224524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.224484    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1addee-ee5f-456d-be31-59caef398724-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.247460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.247437    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rlwq\" (UniqueName: \"kubernetes.io/projected/8e1addee-ee5f-456d-be31-59caef398724-kube-api-access-9rlwq\") pod \"node-exporter-jwcrl\" (UID: \"8e1addee-ee5f-456d-be31-59caef398724\") " pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.386806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.386732    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jwcrl"
Nov 02 12:27:05.398808 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:27:05.398778    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1addee_ee5f_456d_be31_59caef398724.slice/crio-90d13de4641ae7474b8853eaf394631db0aab1de2f85463aad1e58a5679d277d WatchSource:0}: Error finding container 90d13de4641ae7474b8853eaf394631db0aab1de2f85463aad1e58a5679d277d: Status 404 returned error can't find the container with id 90d13de4641ae7474b8853eaf394631db0aab1de2f85463aad1e58a5679d277d
Nov 02 12:27:05.664060 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:05.663976    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwcrl" event={"ID":"8e1addee-ee5f-456d-be31-59caef398724","Type":"ContainerStarted","Data":"90d13de4641ae7474b8853eaf394631db0aab1de2f85463aad1e58a5679d277d"}
Nov 02 12:27:07.432778 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:07.432746    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:27:07.677222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:07.677186    4928 generic.go:356] "Generic (PLEG): container finished" podID="8e1addee-ee5f-456d-be31-59caef398724" containerID="459b4a823c964068b521606aaa85abc295138b1ec32b224c94d5ebc6d7dd69b3" exitCode=0
Nov 02 12:27:07.677368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:07.677231    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwcrl" event={"ID":"8e1addee-ee5f-456d-be31-59caef398724","Type":"ContainerDied","Data":"459b4a823c964068b521606aaa85abc295138b1ec32b224c94d5ebc6d7dd69b3"}
Nov 02 12:27:08.683334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:08.683298    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwcrl" event={"ID":"8e1addee-ee5f-456d-be31-59caef398724","Type":"ContainerStarted","Data":"4c012b589ff886d6960d1b51914f5ace78c19c3e8f684d5885fa8097a510ee0e"}
Nov 02 12:27:08.683720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:08.683355    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwcrl" event={"ID":"8e1addee-ee5f-456d-be31-59caef398724","Type":"ContainerStarted","Data":"646e2def880627775bcc2a0848f5b4f2b2f3d077b7b17b91efe64048c091e4b6"}
Nov 02 12:27:08.776034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:08.775979    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jwcrl" podStartSLOduration=2.359137529 podStartE2EDuration="3.775965981s" totalImagesPullingTime="1.341138112s" totalInitContainerRuntime="75.69034ms" isStatefulPod=true podCreationTimestamp="2026-11-02 12:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:08.774171921 +0000 UTC m=+26.423225879" watchObservedRunningTime="2026-11-02 12:27:08.775965981 +0000 UTC m=+26.425019970"
Nov 02 12:27:09.730280 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:27:09.730241    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:27:11.226765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:11.226735    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:27:16.081277 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:16.081241    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bx8dr"
Nov 02 12:27:20.097317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:20.097283    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:27:22.452216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.452173    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87" podUID="3001af94-ab6c-4995-b400-edecd5c02f59" containerName="registry" containerID="cri-o://862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4" gracePeriod=30
Nov 02 12:27:22.711194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.711118    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:27:22.738928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.738893    4928 generic.go:356] "Generic (PLEG): container finished" podID="3001af94-ab6c-4995-b400-edecd5c02f59" containerID="862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4" exitCode=0
Nov 02 12:27:22.739116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.738940    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87"
Nov 02 12:27:22.739116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.738979    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87" event={"ID":"3001af94-ab6c-4995-b400-edecd5c02f59","Type":"ContainerDied","Data":"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"}
Nov 02 12:27:22.739116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.739066    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846c5f68d8-tjd87" event={"ID":"3001af94-ab6c-4995-b400-edecd5c02f59","Type":"ContainerDied","Data":"096082a01425ca7052f20f695fd293158cf8a3c8b761d8cbc277b999fd3378d8"}
Nov 02 12:27:22.739116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.739090    4928 scope.go:122] "RemoveContainer" containerID="862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"
Nov 02 12:27:22.747364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.747347    4928 scope.go:122] "RemoveContainer" containerID="862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"
Nov 02 12:27:22.747666 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:27:22.747645    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4\": container with ID starting with 862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4 not found: ID does not exist" containerID="862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"
Nov 02 12:27:22.747715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.747676    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4"} err="failed to get container status \"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4\": rpc error: code = NotFound desc = could not find container \"862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4\": container with ID starting with 862d37fda37ccb2d068d9acc4e41fe40ea0738ef846fdd121eeb3aff39a595c4 not found: ID does not exist"
Nov 02 12:27:22.768735 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768710    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.768855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768746    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.768855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768770    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.768855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768810    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.768855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768849    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.769057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768873    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.769057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768902    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.769057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.768946    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") pod \"3001af94-ab6c-4995-b400-edecd5c02f59\" (UID: \"3001af94-ab6c-4995-b400-edecd5c02f59\") "
Nov 02 12:27:22.769210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.769133    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:22.769539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.769512    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:22.771344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.771321    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:22.771462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.771378    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:22.771701 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.771674    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:22.771750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.771720    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "kube-api-access-72k2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:22.771889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.771870    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:22.776393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.776325    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted" pod "3001af94-ab6c-4995-b400-edecd5c02f59" (UID: "3001af94-ab6c-4995-b400-edecd5c02f59"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:22.870219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870185    4928 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-image-registry-private-configuration\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870215    4928 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-registry-certificates\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870228    4928 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3001af94-ab6c-4995-b400-edecd5c02f59-installation-pull-secrets\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870238    4928 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3001af94-ab6c-4995-b400-edecd5c02f59-trusted-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870247    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72k2k\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-kube-api-access-72k2k\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870256    4928 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-registry-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870264    4928 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3001af94-ab6c-4995-b400-edecd5c02f59-ca-trust-extracted\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:22.870446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:22.870272    4928 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3001af94-ab6c-4995-b400-edecd5c02f59-bound-sa-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:23.085947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:23.085916    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:27:23.094177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:23.094150    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-846c5f68d8-tjd87"]
Nov 02 12:27:24.489936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.489862    4928 scope.go:122] "RemoveContainer" containerID="670de40058ce9c565bde29aea575ad05afaee723bd9c48123f68b1d459c1a6d1"
Nov 02 12:27:24.494688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.494663    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3001af94-ab6c-4995-b400-edecd5c02f59" path="/var/lib/kubelet/pods/3001af94-ab6c-4995-b400-edecd5c02f59/volumes"
Nov 02 12:27:24.749400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.749324    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:27:24.749529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.749416    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk" event={"ID":"8e2649ce-6913-40b6-a47a-cd71b5208dd3","Type":"ContainerStarted","Data":"e6b027b5cf173ee143a152f711604b02624240c46988a0e50d5d37005439b780"}
Nov 02 12:27:24.749720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.749700    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:27:24.992725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:24.992691    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-55ghk"
Nov 02 12:27:39.727396 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:27:39.727352    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:27:45.120908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.120821    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry" containerID="cri-o://b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794" gracePeriod=30
Nov 02 12:27:45.367601 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.367576    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:27:45.549774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549723    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.549774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549781    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549819    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549864    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549901    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549925    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.549961    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.550022    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") pod \"532cb38b-3ea7-433c-88de-fb41dba50795\" (UID: \"532cb38b-3ea7-433c-88de-fb41dba50795\") "
Nov 02 12:27:45.550853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.550803    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:45.551022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.550910    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:45.552667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.552633    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:45.552800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.552757    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "kube-api-access-nzhxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:45.552800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.552782    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:45.552939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.552817    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:45.553088 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.553070    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:45.557252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.557228    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted" pod "532cb38b-3ea7-433c-88de-fb41dba50795" (UID: "532cb38b-3ea7-433c-88de-fb41dba50795"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:45.651428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651382    4928 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-installation-pull-secrets\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651423    4928 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-bound-sa-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651440    4928 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/532cb38b-3ea7-433c-88de-fb41dba50795-image-registry-private-configuration\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651455    4928 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-registry-certificates\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651467    4928 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/532cb38b-3ea7-433c-88de-fb41dba50795-trusted-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651479    4928 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/532cb38b-3ea7-433c-88de-fb41dba50795-ca-trust-extracted\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651490    4928 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-registry-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.651612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.651501    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzhxn\" (UniqueName: \"kubernetes.io/projected/532cb38b-3ea7-433c-88de-fb41dba50795-kube-api-access-nzhxn\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:27:45.827101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.827008    4928 generic.go:356] "Generic (PLEG): container finished" podID="532cb38b-3ea7-433c-88de-fb41dba50795" containerID="b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794" exitCode=0
Nov 02 12:27:45.827101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.827060    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7"
Nov 02 12:27:45.827101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.827083    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" event={"ID":"532cb38b-3ea7-433c-88de-fb41dba50795","Type":"ContainerDied","Data":"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"}
Nov 02 12:27:45.827347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.827133    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78fdc47b4f-dp8t7" event={"ID":"532cb38b-3ea7-433c-88de-fb41dba50795","Type":"ContainerDied","Data":"3c627f5d1f4757ef2bac846cb516f0e41e9f92ef38d8ed737f3155a23c5348f4"}
Nov 02 12:27:45.827347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.827160    4928 scope.go:122] "RemoveContainer" containerID="b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"
Nov 02 12:27:45.835940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.835915    4928 scope.go:122] "RemoveContainer" containerID="b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"
Nov 02 12:27:45.836208 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:27:45.836189    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794\": container with ID starting with b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794 not found: ID does not exist" containerID="b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"
Nov 02 12:27:45.836255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.836218    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794"} err="failed to get container status \"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794\": rpc error: code = NotFound desc = could not find container \"b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794\": container with ID starting with b05601e48b9b6c41f3b415d2d5cf246d8798c4f32a1d910f08ad23312690b794 not found: ID does not exist"
Nov 02 12:27:45.880796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.880765    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:27:45.894261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:45.894233    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78fdc47b4f-dp8t7"]
Nov 02 12:27:46.495051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:46.495019    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" path="/var/lib/kubelet/pods/532cb38b-3ea7-433c-88de-fb41dba50795/volumes"
Nov 02 12:27:47.836004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:47.835967    4928 generic.go:356] "Generic (PLEG): container finished" podID="10ae75b6-203e-4d60-b728-412645e0a4a8" containerID="84d44cda47bc7b26379c51e7aa2ba9b51c34b36a0aabeaaffd5e518fe5e35938" exitCode=0
Nov 02 12:27:47.836397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:47.836042    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl" event={"ID":"10ae75b6-203e-4d60-b728-412645e0a4a8","Type":"ContainerDied","Data":"84d44cda47bc7b26379c51e7aa2ba9b51c34b36a0aabeaaffd5e518fe5e35938"}
Nov 02 12:27:47.836440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:47.836422    4928 scope.go:122] "RemoveContainer" containerID="84d44cda47bc7b26379c51e7aa2ba9b51c34b36a0aabeaaffd5e518fe5e35938"
Nov 02 12:27:48.841361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:27:48.841330    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2l2kl" event={"ID":"10ae75b6-203e-4d60-b728-412645e0a4a8","Type":"ContainerStarted","Data":"18646f2202e809f54a20ba26b55c552345a43e307cfa4fd9086987b9065e887a"}
Nov 02 12:28:07.911547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:07.911513    4928 generic.go:356] "Generic (PLEG): container finished" podID="0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151" containerID="90ebfa19ebc3d970b0e6e33bb927f3d39252b2d420e2972b1831c32fef7e4214" exitCode=0
Nov 02 12:28:07.912032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:07.911587    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4" event={"ID":"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151","Type":"ContainerDied","Data":"90ebfa19ebc3d970b0e6e33bb927f3d39252b2d420e2972b1831c32fef7e4214"}
Nov 02 12:28:07.912032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:07.911941    4928 scope.go:122] "RemoveContainer" containerID="90ebfa19ebc3d970b0e6e33bb927f3d39252b2d420e2972b1831c32fef7e4214"
Nov 02 12:28:08.917053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:08.917015    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gj2h4" event={"ID":"0d5dcaa3-4ad1-4586-ad24-b8d7bb74e151","Type":"ContainerStarted","Data":"fe54334454c29cc9da8778bc333e4f3cc6fca672c41f0e89449a770ad063882d"}
Nov 02 12:28:09.728104 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:28:09.728066    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:28:12.933292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:12.933254    4928 generic.go:356] "Generic (PLEG): container finished" podID="6da4be61-ac40-4531-9357-e05f84219250" containerID="045f12b705abbcb2709ea1b6807dd0e9facac15e4475bc08e687eedc42124cd9" exitCode=0
Nov 02 12:28:12.933782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:12.933325    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-x6zzd" event={"ID":"6da4be61-ac40-4531-9357-e05f84219250","Type":"ContainerDied","Data":"045f12b705abbcb2709ea1b6807dd0e9facac15e4475bc08e687eedc42124cd9"}
Nov 02 12:28:12.933782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:12.933714    4928 scope.go:122] "RemoveContainer" containerID="045f12b705abbcb2709ea1b6807dd0e9facac15e4475bc08e687eedc42124cd9"
Nov 02 12:28:13.938128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:28:13.938094    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-x6zzd" event={"ID":"6da4be61-ac40-4531-9357-e05f84219250","Type":"ContainerStarted","Data":"aea635a8c956beffcdecf09d0657a2d2de9e53ba8963758c469cfa4babbc4ebc"}
Nov 02 12:28:39.727624 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:28:39.727581    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:29:09.726544 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:29:09.726465    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:29:29.164526 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164490    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"]
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164784    4928 cpu_manager.go:414] "containerMap: removing container" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry"
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164797    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry"
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164843    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3001af94-ab6c-4995-b400-edecd5c02f59" containerName="registry"
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164852    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3001af94-ab6c-4995-b400-edecd5c02f59" containerName="registry"
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164953    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3001af94-ab6c-4995-b400-edecd5c02f59" containerName="registry"
Nov 02 12:29:29.164973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.164968    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="532cb38b-3ea7-433c-88de-fb41dba50795" containerName="registry"
Nov 02 12:29:29.168151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.168131    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.207674 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.207640    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 12:29:29.207885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.207728    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 12:29:29.207885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.207730    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zjdz9\""
Nov 02 12:29:29.208514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.208492    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"]
Nov 02 12:29:29.226297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.226265    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwt8k\" (UniqueName: \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.226414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.226310    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.226414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.226337    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.304449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.304412    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"]
Nov 02 12:29:29.310839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.310800    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.326751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326723    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.326920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326757    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.326920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326779    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwt8k\" (UniqueName: \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.326920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326887    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.326920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326910    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.327150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.326947    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22m4\" (UniqueName: \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.327226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.327208    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.327264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.327232    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.333181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.333154    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q4jp5\""
Nov 02 12:29:29.334360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.334335    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"]
Nov 02 12:29:29.352379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.352349    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwt8k\" (UniqueName: \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.428142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.428062    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.428142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.428097    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.428142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.428131    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22m4\" (UniqueName: \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.428444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.428423    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.428482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.428466    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.452635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.452596    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22m4\" (UniqueName: \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.483343 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.483314    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:29:29.627018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.626978    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:29:29.783685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.783651    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"]
Nov 02 12:29:29.786673 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:29.786647    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97f9058_0a18_495c_a764_2bacf1c207b3.slice/crio-b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836 WatchSource:0}: Error finding container b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836: Status 404 returned error can't find the container with id b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836
Nov 02 12:29:29.923024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:29.922979    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"]
Nov 02 12:29:29.926511 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:29.926481    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d77be_ea00_4c51_9da1_d2ad18cd97e0.slice/crio-b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037 WatchSource:0}: Error finding container b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037: Status 404 returned error can't find the container with id b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037
Nov 02 12:29:30.213330 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:30.213244    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerStarted","Data":"b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037"}
Nov 02 12:29:30.214746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:30.214715    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b" event={"ID":"a97f9058-0a18-495c-a764-2bacf1c207b3","Type":"ContainerStarted","Data":"b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836"}
Nov 02 12:29:32.107790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.107740    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"]
Nov 02 12:29:32.112353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.111729    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.141273 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.141227    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-c78bl\""
Nov 02 12:29:32.142483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.142457    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"]
Nov 02 12:29:32.151972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.151875    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.151972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.151948    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.152149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.152090    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgsp\" (UniqueName: \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.253275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.253237    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgsp\" (UniqueName: \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.253458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.253283    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.253458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.253339    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.253819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.253795    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.254090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.254070    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.278338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.278302    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgsp\" (UniqueName: \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.431925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.431841    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:29:32.746652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:32.746616    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"]
Nov 02 12:29:32.753004 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:32.752943    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87be9f36_afe8_4b0a_9e9e_55489a558bbd.slice/crio-a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9 WatchSource:0}: Error finding container a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9: Status 404 returned error can't find the container with id a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9
Nov 02 12:29:33.231478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.231390    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2" event={"ID":"87be9f36-afe8-4b0a-9e9e-55489a558bbd","Type":"ContainerStarted","Data":"a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9"}
Nov 02 12:29:33.281435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.281402    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"]
Nov 02 12:29:33.285340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.285317    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.316143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.316110    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"]
Nov 02 12:29:33.316298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.316246    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dr4zz\""
Nov 02 12:29:33.362257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.361976    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pr6\" (UniqueName: \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.362257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.362036    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.362257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.362072    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.380213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.380179    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"]
Nov 02 12:29:33.384243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.384190    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.405028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.404985    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-6twtf\""
Nov 02 12:29:33.406282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.406238    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"]
Nov 02 12:29:33.462732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462685    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.462949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462757    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pr6\" (UniqueName: \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.462949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462794    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.462949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462818    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.462949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462878    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vghr\" (UniqueName: \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.462949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.462926    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.463705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.463675    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.463986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.463965    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.477568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.477538    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"]
Nov 02 12:29:33.481668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.481612    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.488888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.488861    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pr6\" (UniqueName: \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.503570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.503540    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"]
Nov 02 12:29:33.503808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.503791    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-brclx\""
Nov 02 12:29:33.563959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.563919    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75sp5\" (UniqueName: \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.564149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.563972    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vghr\" (UniqueName: \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.564149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.564064    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.564149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.564129    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.564318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.564169    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.564318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.564230    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.566166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.566133    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.566639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.566605    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.578329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.578304    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"]
Nov 02 12:29:33.582468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.582444    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.590244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.590168    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vghr\" (UniqueName: \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.606897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.602670    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7rpmg\""
Nov 02 12:29:33.606897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.603643    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"]
Nov 02 12:29:33.607336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.607314    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:29:33.665720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665681    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.665929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665743    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.665929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665779    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.665929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665816    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75sp5\" (UniqueName: \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.665929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665902    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.666144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.665930    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjw4\" (UniqueName: \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.666573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.666546    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.666768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.666723    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.698306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.698277    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75sp5\" (UniqueName: \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.703178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.702993    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:29:33.767376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.766438    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.767376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.766484    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.767376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.766504    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjw4\" (UniqueName: \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.767376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.767153    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.770481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.769871    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.790920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.790885    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjw4\" (UniqueName: \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.814765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.814533    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:29:33.915093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.915053    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:29:33.927135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:33.927049    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"]
Nov 02 12:29:33.930333 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:33.930209    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbdb3b2_47e8_42a3_8fed_f6199908778f.slice/crio-2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c WatchSource:0}: Error finding container 2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c: Status 404 returned error can't find the container with id 2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c
Nov 02 12:29:34.021463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.021425    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"]
Nov 02 12:29:34.025246 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:34.025206    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423dc021_9142_468f_98bc_174b8a50e096.slice/crio-ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b WatchSource:0}: Error finding container ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b: Status 404 returned error can't find the container with id ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b
Nov 02 12:29:34.158612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.158586    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"]
Nov 02 12:29:34.161898 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:34.161866    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb106336e_aee7_4e00_bd14_0cd9ddf5da90.slice/crio-93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305 WatchSource:0}: Error finding container 93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305: Status 404 returned error can't find the container with id 93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305
Nov 02 12:29:34.234278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.234243    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"]
Nov 02 12:29:34.240041 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:34.239999    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bec8dc_7794_489b_ab4e_64455c2f5a70.slice/crio-f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b WatchSource:0}: Error finding container f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b: Status 404 returned error can't find the container with id f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b
Nov 02 12:29:34.244626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.244594    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerStarted","Data":"93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305"}
Nov 02 12:29:34.246522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.246498    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerStarted","Data":"ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b"}
Nov 02 12:29:34.247792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.247765    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr" event={"ID":"1cbdb3b2-47e8-42a3-8fed-f6199908778f","Type":"ContainerStarted","Data":"2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c"}
Nov 02 12:29:34.947673 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.947628    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:29:34.951866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.951821    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.969359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.968164    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 12:29:34.969766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.969726    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2nvrx\""
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978127    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978163    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978195    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkjv\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978233    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978302    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978334    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978366    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978410    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.978629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.978435    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:34.982451 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:34.982430    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:29:35.077609 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.077006    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"]
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079315    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079358    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079392    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079435    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079462    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079513    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079538    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079566    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkjv\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.079604    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.080646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.080261    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.082357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.081564    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.082357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.081647    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.082357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.082278    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.082357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.082328    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.084605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.083593    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.087286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.087246    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.089847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.089781    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.116803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.105255    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-jp6pj\""
Nov 02 12:29:35.116803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.108339    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"]
Nov 02 12:29:35.133522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.133447    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkjv\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.135588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.135522    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2x5c4\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180849    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180896    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea80c1c-f77d-4271-a069-73021aaae094-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180928    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllsw\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-kube-api-access-sllsw\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180947    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180965    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea80c1c-f77d-4271-a069-73021aaae094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.180995    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.181028    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.181051    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.181238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.181071    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.258102 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.258059    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerStarted","Data":"f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b"}
Nov 02 12:29:35.280955 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.280604    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281554    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllsw\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-kube-api-access-sllsw\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281610    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281638    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea80c1c-f77d-4271-a069-73021aaae094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281690    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281744    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.281782    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282679 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.282344    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.282872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.282753    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea80c1c-f77d-4271-a069-73021aaae094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283012    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283268    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283340    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283366    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea80c1c-f77d-4271-a069-73021aaae094-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283435    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.283710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.283670    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.294292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.294236    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea80c1c-f77d-4271-a069-73021aaae094-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.302163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.302139    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea80c1c-f77d-4271-a069-73021aaae094-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.335407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.335310    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllsw\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-kube-api-access-sllsw\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.346144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.346082    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea80c1c-f77d-4271-a069-73021aaae094-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth\" (UID: \"fea80c1c-f77d-4271-a069-73021aaae094\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.428517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.428195    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:35.618134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.617305    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"]
Nov 02 12:29:35.624254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.624218    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.657669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.657564    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"]
Nov 02 12:29:35.659331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.659215    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:29:35.686906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.686678    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.686906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.686720    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4ls\" (UniqueName: \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.686906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.686790    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.782688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.782603    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"]
Nov 02 12:29:35.787581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.787548    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.787691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.787642    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.787691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.787673    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4ls\" (UniqueName: \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.788362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.788333    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.788654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.788628    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.792643 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:35.792617    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea80c1c_f77d_4271_a069_73021aaae094.slice/crio-22e139a78727db0c0cef6219188d78055eb0856a78928dd15b32988363cac606 WatchSource:0}: Error finding container 22e139a78727db0c0cef6219188d78055eb0856a78928dd15b32988363cac606: Status 404 returned error can't find the container with id 22e139a78727db0c0cef6219188d78055eb0856a78928dd15b32988363cac606
Nov 02 12:29:35.812690 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.812661    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4ls\" (UniqueName: \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:35.967086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:35.967052    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:29:36.272423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:36.272380    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth" event={"ID":"fea80c1c-f77d-4271-a069-73021aaae094","Type":"ContainerStarted","Data":"22e139a78727db0c0cef6219188d78055eb0856a78928dd15b32988363cac606"}
Nov 02 12:29:36.279178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:36.279142    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerStarted","Data":"54af0422d0c7fc75c535a900b3a79f8664442cc29921897eef6529d40ffaa627"}
Nov 02 12:29:36.315752 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:36.315672    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"]
Nov 02 12:29:36.325459 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:29:36.325423    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac81d82_7f9e_42b2_9cdd_c2c33d3ee6bd.slice/crio-03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8 WatchSource:0}: Error finding container 03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8: Status 404 returned error can't find the container with id 03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8
Nov 02 12:29:37.313784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:37.313705    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh" event={"ID":"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd","Type":"ContainerStarted","Data":"03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8"}
Nov 02 12:29:39.787256 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:29:39.787162    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:29:42.377689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:42.376883    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerStarted","Data":"24249fe3dfb74e13c6b07ab88ea69314f2299522139e239d5a3343c6d18034d0"}
Nov 02 12:29:43.394953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.394909    4928 generic.go:356] "Generic (PLEG): container finished" podID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerID="24249fe3dfb74e13c6b07ab88ea69314f2299522139e239d5a3343c6d18034d0" exitCode=0
Nov 02 12:29:43.395428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.395014    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerDied","Data":"24249fe3dfb74e13c6b07ab88ea69314f2299522139e239d5a3343c6d18034d0"}
Nov 02 12:29:43.401252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.401223    4928 generic.go:356] "Generic (PLEG): container finished" podID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerID="446ee87482d55d38357075016fd9b9e863cf0e731ca498dbbb93a7f50870fa99" exitCode=0
Nov 02 12:29:43.401339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.401311    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b" event={"ID":"a97f9058-0a18-495c-a764-2bacf1c207b3","Type":"ContainerDied","Data":"446ee87482d55d38357075016fd9b9e863cf0e731ca498dbbb93a7f50870fa99"}
Nov 02 12:29:43.888687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.888643    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:43.888812 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.888741    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:43.888812 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:43.888773    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:44.321858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:44.321562    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:44.321858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:44.321684    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:44.321858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:44.321729    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:29:44.411125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:44.411095    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerStarted","Data":"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"}
Nov 02 12:29:44.462412 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:44.462017    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podStartSLOduration=2.242327946 podStartE2EDuration="10.461998042s" totalImagesPullingTime="8.219670096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:44.458416604 +0000 UTC m=+182.107470568" watchObservedRunningTime="2026-11-02 12:29:44.461998042 +0000 UTC m=+182.111052002"
Nov 02 12:29:45.281212 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.281178    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:45.287766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.287734    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:45.420724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.420646    4928 generic.go:356] "Generic (PLEG): container finished" podID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerID="9d2ad00cf79ee5614620bf09bc9a6517eb111625af5afbdce48384fb20cfecd8" exitCode=0
Nov 02 12:29:45.421148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.420880    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerDied","Data":"9d2ad00cf79ee5614620bf09bc9a6517eb111625af5afbdce48384fb20cfecd8"}
Nov 02 12:29:45.424558 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.424534    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth" event={"ID":"fea80c1c-f77d-4271-a069-73021aaae094","Type":"ContainerStarted","Data":"6d596665b41d1dc6f9dc32bc584234260d1b3f640ad40536fc60df306037ac8e"}
Nov 02 12:29:45.427804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.427780    4928 generic.go:356] "Generic (PLEG): container finished" podID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerID="1cf09a9d4ce09374943792aaf6ca8b6534182bb4e215081798c2f2563351e3f1" exitCode=0
Nov 02 12:29:45.427961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.427942    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerDied","Data":"1cf09a9d4ce09374943792aaf6ca8b6534182bb4e215081798c2f2563351e3f1"}
Nov 02 12:29:45.428680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.428670    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:45.434606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.434581    4928 generic.go:356] "Generic (PLEG): container finished" podID="423dc021-9142-468f-98bc-174b8a50e096" containerID="2e1d0a5e0e72a72c0a65b7273cf7a05a92a36ec3ca7522830d8940ed4a8fa829" exitCode=0
Nov 02 12:29:45.434719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.434701    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerDied","Data":"2e1d0a5e0e72a72c0a65b7273cf7a05a92a36ec3ca7522830d8940ed4a8fa829"}
Nov 02 12:29:45.435506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.435488    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:45.440407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.440379    4928 generic.go:356] "Generic (PLEG): container finished" podID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerID="79b1ae145505dc3aa49a24bbb264e3ed1d5ab26b82d8e5f30e1140900423d40f" exitCode=0
Nov 02 12:29:45.441176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.441148    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr" event={"ID":"1cbdb3b2-47e8-42a3-8fed-f6199908778f","Type":"ContainerDied","Data":"79b1ae145505dc3aa49a24bbb264e3ed1d5ab26b82d8e5f30e1140900423d40f"}
Nov 02 12:29:45.441273 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.441202    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:45.443546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.443515    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:29:45.606733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:45.606672    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth" podStartSLOduration=2.081915539 podStartE2EDuration="10.606651692s" totalImagesPullingTime="8.524736153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:45.604554267 +0000 UTC m=+183.253608226" watchObservedRunningTime="2026-11-02 12:29:45.606651692 +0000 UTC m=+183.255705652"
Nov 02 12:29:46.450715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.450082    4928 generic.go:356] "Generic (PLEG): container finished" podID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerID="dac7e64714eeba0f87a952c0606f7a094dad23ebd8af4d835fc601165718ab27" exitCode=0
Nov 02 12:29:46.450715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.450220    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerDied","Data":"dac7e64714eeba0f87a952c0606f7a094dad23ebd8af4d835fc601165718ab27"}
Nov 02 12:29:46.457000 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.453437    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerStarted","Data":"e245e5ee8487a933f104bd5bd84bdf77e6884a6188bbf50a44fcbf9b777e65ad"}
Nov 02 12:29:46.458246 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.458218    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerStarted","Data":"54c9d89816269ff691a99774396dffc2f6935e4b9be9e41c704fdbb2b0e0f359"}
Nov 02 12:29:46.458480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.458437    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:46.459940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:46.459916    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth"
Nov 02 12:29:47.466051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.466023    4928 generic.go:356] "Generic (PLEG): container finished" podID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerID="e17b4021f9ccd217af9a1eda000ce1fb183e4e92bc99b7806d76a73757674ab2" exitCode=0
Nov 02 12:29:47.466429 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.466111    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerDied","Data":"e17b4021f9ccd217af9a1eda000ce1fb183e4e92bc99b7806d76a73757674ab2"}
Nov 02 12:29:47.469294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.469266    4928 generic.go:356] "Generic (PLEG): container finished" podID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerID="4a96c597c5f38dad96209d63b1faf4bbf689824d041e1bd7f9a6da42eb67f429" exitCode=0
Nov 02 12:29:47.469397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.469358    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b" event={"ID":"a97f9058-0a18-495c-a764-2bacf1c207b3","Type":"ContainerDied","Data":"4a96c597c5f38dad96209d63b1faf4bbf689824d041e1bd7f9a6da42eb67f429"}
Nov 02 12:29:47.473641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.472639    4928 generic.go:356] "Generic (PLEG): container finished" podID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerID="ca8517764f78bf52a99fdeee8727d274bfb099a5676d2c4d4619960b9cc465e1" exitCode=0
Nov 02 12:29:47.473641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.472790    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr" event={"ID":"1cbdb3b2-47e8-42a3-8fed-f6199908778f","Type":"ContainerDied","Data":"ca8517764f78bf52a99fdeee8727d274bfb099a5676d2c4d4619960b9cc465e1"}
Nov 02 12:29:47.475359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.475330    4928 generic.go:356] "Generic (PLEG): container finished" podID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerID="bfca24b5a99f26cb90c5370b91ebebeec0d5386672420477ff126af85638b143" exitCode=0
Nov 02 12:29:47.475473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.475450    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh" event={"ID":"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd","Type":"ContainerDied","Data":"bfca24b5a99f26cb90c5370b91ebebeec0d5386672420477ff126af85638b143"}
Nov 02 12:29:47.479992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.479771    4928 generic.go:356] "Generic (PLEG): container finished" podID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerID="e245e5ee8487a933f104bd5bd84bdf77e6884a6188bbf50a44fcbf9b777e65ad" exitCode=0
Nov 02 12:29:47.479992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.479819    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerDied","Data":"e245e5ee8487a933f104bd5bd84bdf77e6884a6188bbf50a44fcbf9b777e65ad"}
Nov 02 12:29:47.484797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.483894    4928 generic.go:356] "Generic (PLEG): container finished" podID="423dc021-9142-468f-98bc-174b8a50e096" containerID="54c9d89816269ff691a99774396dffc2f6935e4b9be9e41c704fdbb2b0e0f359" exitCode=0
Nov 02 12:29:47.484797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:47.484308    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerDied","Data":"54c9d89816269ff691a99774396dffc2f6935e4b9be9e41c704fdbb2b0e0f359"}
Nov 02 12:29:48.499545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:48.497460    4928 generic.go:356] "Generic (PLEG): container finished" podID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerID="f1995c0974ab8fff320130e57504dfbcd603935d9a9e36ef96163b6adf6c0cb9" exitCode=0
Nov 02 12:29:48.502720 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:48.502642    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2" event={"ID":"87be9f36-afe8-4b0a-9e9e-55489a558bbd","Type":"ContainerDied","Data":"f1995c0974ab8fff320130e57504dfbcd603935d9a9e36ef96163b6adf6c0cb9"}
Nov 02 12:29:48.508257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:48.506615    4928 generic.go:356] "Generic (PLEG): container finished" podID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerID="600bb0433446595b8e10f6be807d14b3ee023231dbfc46942dff6a3b7df9801e" exitCode=0
Nov 02 12:29:48.508257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:48.508196    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh" event={"ID":"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd","Type":"ContainerDied","Data":"600bb0433446595b8e10f6be807d14b3ee023231dbfc46942dff6a3b7df9801e"}
Nov 02 12:29:49.519460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:49.518241    4928 generic.go:356] "Generic (PLEG): container finished" podID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerID="67d3ae7bb6eb4c7ee1e23c793cb08f3684b8cb24258baef6dbeb474deac0b73d" exitCode=0
Nov 02 12:29:49.519460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:29:49.518398    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2" event={"ID":"87be9f36-afe8-4b0a-9e9e-55489a558bbd","Type":"ContainerDied","Data":"67d3ae7bb6eb4c7ee1e23c793cb08f3684b8cb24258baef6dbeb474deac0b73d"}
Nov 02 12:30:02.675703 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:02.675661    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerStarted","Data":"e54748b69ee47a84c6bb74aa1c7333e1458d25c395324fcc0dd36aba2c86d3c0"}
Nov 02 12:30:02.680800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:02.680543    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerStarted","Data":"496146fb95ffe58f0f3d9c4d4d9d94eb2fd339a069a643ea231975878909f7f5"}
Nov 02 12:30:02.768067 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:02.767441    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" podStartSLOduration=2.651483119 podStartE2EDuration="29.767424725s" totalImagesPullingTime="27.115941606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:02.722246713 +0000 UTC m=+200.371300672" watchObservedRunningTime="2026-11-02 12:30:02.767424725 +0000 UTC m=+200.416478684"
Nov 02 12:30:03.692236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.691963    4928 generic.go:356] "Generic (PLEG): container finished" podID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerID="e54748b69ee47a84c6bb74aa1c7333e1458d25c395324fcc0dd36aba2c86d3c0" exitCode=0
Nov 02 12:30:03.692752 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.692437    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerDied","Data":"e54748b69ee47a84c6bb74aa1c7333e1458d25c395324fcc0dd36aba2c86d3c0"}
Nov 02 12:30:03.695156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.695127    4928 generic.go:356] "Generic (PLEG): container finished" podID="423dc021-9142-468f-98bc-174b8a50e096" containerID="496146fb95ffe58f0f3d9c4d4d9d94eb2fd339a069a643ea231975878909f7f5" exitCode=0
Nov 02 12:30:03.695297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.695217    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerDied","Data":"496146fb95ffe58f0f3d9c4d4d9d94eb2fd339a069a643ea231975878909f7f5"}
Nov 02 12:30:03.698225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.698195    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerStarted","Data":"a76d80e8eb3ad59e759591beceb6672812e7f9cfce8486a674628e5086638749"}
Nov 02 12:30:03.787630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.787475    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" podStartSLOduration=2.750475891 podStartE2EDuration="34.787460552s" totalImagesPullingTime="32.036984661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:03.785608269 +0000 UTC m=+201.434662229" watchObservedRunningTime="2026-11-02 12:30:03.787460552 +0000 UTC m=+201.436514510"
Nov 02 12:30:03.863996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.861429    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:30:03.987358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.987333    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\" (UniqueName: \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\") pod \"423dc021-9142-468f-98bc-174b8a50e096\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") "
Nov 02 12:30:03.987904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.987468    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\") pod \"423dc021-9142-468f-98bc-174b8a50e096\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") "
Nov 02 12:30:03.987904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.987509    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\") pod \"423dc021-9142-468f-98bc-174b8a50e096\" (UID: \"423dc021-9142-468f-98bc-174b8a50e096\") "
Nov 02 12:30:03.988322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.988296    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle" pod "423dc021-9142-468f-98bc-174b8a50e096" (UID: "423dc021-9142-468f-98bc-174b8a50e096"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:03.990427 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.990407    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr" pod "423dc021-9142-468f-98bc-174b8a50e096" (UID: "423dc021-9142-468f-98bc-174b8a50e096"). InnerVolumeSpecName "kube-api-access-6vghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:03.993901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:03.993875    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util" pod "423dc021-9142-468f-98bc-174b8a50e096" (UID: "423dc021-9142-468f-98bc-174b8a50e096"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:04.088510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.088469    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vghr\" (UniqueName: \"kubernetes.io/projected/423dc021-9142-468f-98bc-174b8a50e096-kube-api-access-6vghr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:04.088629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.088517    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:04.088629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.088540    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/423dc021-9142-468f-98bc-174b8a50e096-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:04.706576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.706493    4928 generic.go:356] "Generic (PLEG): container finished" podID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerID="9e6aedc84dab45f73b0a1c729de5612944675f77fad4595dbd0936c245a86339" exitCode=0
Nov 02 12:30:04.707040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.706627    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerDied","Data":"9e6aedc84dab45f73b0a1c729de5612944675f77fad4595dbd0936c245a86339"}
Nov 02 12:30:04.709185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.709155    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd" event={"ID":"423dc021-9142-468f-98bc-174b8a50e096","Type":"ContainerDied","Data":"ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b"}
Nov 02 12:30:04.709316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.709215    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef08ce5161f069a036be6d915d2139c30a8b5ef16cf2a706e9bce9f134869e5b"
Nov 02 12:30:04.709381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.709318    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7rcd"
Nov 02 12:30:04.714974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.714942    4928 generic.go:356] "Generic (PLEG): container finished" podID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerID="a76d80e8eb3ad59e759591beceb6672812e7f9cfce8486a674628e5086638749" exitCode=0
Nov 02 12:30:04.715124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.715027    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerDied","Data":"a76d80e8eb3ad59e759591beceb6672812e7f9cfce8486a674628e5086638749"}
Nov 02 12:30:04.717782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.717753    4928 generic.go:356] "Generic (PLEG): container finished" podID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerID="76e0eba6ed245a0bf4c33d2491f0168bb19e501d32a7b7dc688769b4e4f74dd3" exitCode=0
Nov 02 12:30:04.717947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.717868    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b" event={"ID":"a97f9058-0a18-495c-a764-2bacf1c207b3","Type":"ContainerDied","Data":"76e0eba6ed245a0bf4c33d2491f0168bb19e501d32a7b7dc688769b4e4f74dd3"}
Nov 02 12:30:04.720366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.720340    4928 generic.go:356] "Generic (PLEG): container finished" podID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerID="cdba4b59d861848c5927de166d2c68b28255f9d8f4de73a94931407fe2de6ab8" exitCode=0
Nov 02 12:30:04.720491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.720396    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr" event={"ID":"1cbdb3b2-47e8-42a3-8fed-f6199908778f","Type":"ContainerDied","Data":"cdba4b59d861848c5927de166d2c68b28255f9d8f4de73a94931407fe2de6ab8"}
Nov 02 12:30:04.878622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.878590    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:30:04.997979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.997943    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\" (UniqueName: \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\") pod \"12bec8dc-7794-489b-ab4e-64455c2f5a70\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") "
Nov 02 12:30:04.998133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.998001    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\") pod \"12bec8dc-7794-489b-ab4e-64455c2f5a70\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") "
Nov 02 12:30:04.998133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.998061    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\") pod \"12bec8dc-7794-489b-ab4e-64455c2f5a70\" (UID: \"12bec8dc-7794-489b-ab4e-64455c2f5a70\") "
Nov 02 12:30:04.999289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:04.999252    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle" pod "12bec8dc-7794-489b-ab4e-64455c2f5a70" (UID: "12bec8dc-7794-489b-ab4e-64455c2f5a70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:05.001110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.001077    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4" pod "12bec8dc-7794-489b-ab4e-64455c2f5a70" (UID: "12bec8dc-7794-489b-ab4e-64455c2f5a70"). InnerVolumeSpecName "kube-api-access-8hjw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:05.004738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.004710    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util" pod "12bec8dc-7794-489b-ab4e-64455c2f5a70" (UID: "12bec8dc-7794-489b-ab4e-64455c2f5a70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:05.099604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.099557    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:05.099604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.099592    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjw4\" (UniqueName: \"kubernetes.io/projected/12bec8dc-7794-489b-ab4e-64455c2f5a70-kube-api-access-8hjw4\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:05.099604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.099604    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bec8dc-7794-489b-ab4e-64455c2f5a70-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:05.728172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.728146    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx"
Nov 02 12:30:05.728611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.728143    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8xfjx" event={"ID":"12bec8dc-7794-489b-ab4e-64455c2f5a70","Type":"ContainerDied","Data":"f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b"}
Nov 02 12:30:05.728611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.728286    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f524513ac27482aa485d154de555f1ce21b9bb63e26a1aa6aee89ae1fbffc66b"
Nov 02 12:30:05.730747 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.730715    4928 generic.go:356] "Generic (PLEG): container finished" podID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerID="5cc80e11286193ad7e8af7fcbfd8e952d125f0212d320826e1873af27340e7f7" exitCode=0
Nov 02 12:30:05.730916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.730761    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh" event={"ID":"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd","Type":"ContainerDied","Data":"5cc80e11286193ad7e8af7fcbfd8e952d125f0212d320826e1873af27340e7f7"}
Nov 02 12:30:05.925063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.925037    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:30:05.949191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.949165    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:30:05.971387 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.971355    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:30:05.986179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:05.986155    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:30:06.109695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109663    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\" (UniqueName: \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\") pod \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") "
Nov 02 12:30:06.109891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109710    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\" (UniqueName: \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\") pod \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") "
Nov 02 12:30:06.109891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109741    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\") pod \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") "
Nov 02 12:30:06.109891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109773    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\") pod \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") "
Nov 02 12:30:06.109891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109818    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\") pod \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") "
Nov 02 12:30:06.109891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109881    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\") pod \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\" (UID: \"b106336e-aee7-4e00-bd14-0cd9ddf5da90\") "
Nov 02 12:30:06.110158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109912    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\") pod \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\" (UID: \"1cbdb3b2-47e8-42a3-8fed-f6199908778f\") "
Nov 02 12:30:06.110158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.109977    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\" (UniqueName: \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\") pod \"a97f9058-0a18-495c-a764-2bacf1c207b3\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") "
Nov 02 12:30:06.110158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.110013    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\" (UniqueName: \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\") pod \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") "
Nov 02 12:30:06.110158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.110052    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\") pod \"a97f9058-0a18-495c-a764-2bacf1c207b3\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") "
Nov 02 12:30:06.110158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.110096    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\") pod \"a97f9058-0a18-495c-a764-2bacf1c207b3\" (UID: \"a97f9058-0a18-495c-a764-2bacf1c207b3\") "
Nov 02 12:30:06.110395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.110168    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\") pod \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\" (UID: \"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0\") "
Nov 02 12:30:06.111292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.110973    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle" pod "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" (UID: "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.111292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.111195    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle" pod "1cbdb3b2-47e8-42a3-8fed-f6199908778f" (UID: "1cbdb3b2-47e8-42a3-8fed-f6199908778f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.111775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.111742    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle" pod "b106336e-aee7-4e00-bd14-0cd9ddf5da90" (UID: "b106336e-aee7-4e00-bd14-0cd9ddf5da90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.111953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.111928    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle" pod "a97f9058-0a18-495c-a764-2bacf1c207b3" (UID: "a97f9058-0a18-495c-a764-2bacf1c207b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.112528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.112495    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6" pod "1cbdb3b2-47e8-42a3-8fed-f6199908778f" (UID: "1cbdb3b2-47e8-42a3-8fed-f6199908778f"). InnerVolumeSpecName "kube-api-access-89pr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:06.113025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.112995    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5" pod "b106336e-aee7-4e00-bd14-0cd9ddf5da90" (UID: "b106336e-aee7-4e00-bd14-0cd9ddf5da90"). InnerVolumeSpecName "kube-api-access-75sp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:06.114266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.114245    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4" pod "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" (UID: "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0"). InnerVolumeSpecName "kube-api-access-t22m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:06.114488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.114472    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k" pod "a97f9058-0a18-495c-a764-2bacf1c207b3" (UID: "a97f9058-0a18-495c-a764-2bacf1c207b3"). InnerVolumeSpecName "kube-api-access-zwt8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:06.118808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.118779    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util" pod "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" (UID: "cc6d77be-ea00-4c51-9da1-d2ad18cd97e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.119029 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.119002    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util" pod "1cbdb3b2-47e8-42a3-8fed-f6199908778f" (UID: "1cbdb3b2-47e8-42a3-8fed-f6199908778f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.119253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.119233    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util" pod "b106336e-aee7-4e00-bd14-0cd9ddf5da90" (UID: "b106336e-aee7-4e00-bd14-0cd9ddf5da90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.119594 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.119572    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util" pod "a97f9058-0a18-495c-a764-2bacf1c207b3" (UID: "a97f9058-0a18-495c-a764-2bacf1c207b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211563    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75sp5\" (UniqueName: \"kubernetes.io/projected/b106336e-aee7-4e00-bd14-0cd9ddf5da90-kube-api-access-75sp5\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211599    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pr6\" (UniqueName: \"kubernetes.io/projected/1cbdb3b2-47e8-42a3-8fed-f6199908778f-kube-api-access-89pr6\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211612    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211625    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211636    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211646    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b106336e-aee7-4e00-bd14-0cd9ddf5da90-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211657    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cbdb3b2-47e8-42a3-8fed-f6199908778f-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.211663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211668    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwt8k\" (UniqueName: \"kubernetes.io/projected/a97f9058-0a18-495c-a764-2bacf1c207b3-kube-api-access-zwt8k\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.212075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211681    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22m4\" (UniqueName: \"kubernetes.io/projected/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-kube-api-access-t22m4\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.212075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211692    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.212075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211703    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a97f9058-0a18-495c-a764-2bacf1c207b3-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.212075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.211713    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc6d77be-ea00-4c51-9da1-d2ad18cd97e0-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:06.737342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.737304    4928 generic.go:356] "Generic (PLEG): container finished" podID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerID="edbd416b5f549490b0f73705dd05eb729fae7334b70cc7b1e075e9ec063a30d8" exitCode=0
Nov 02 12:30:06.737848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.737387    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2" event={"ID":"87be9f36-afe8-4b0a-9e9e-55489a558bbd","Type":"ContainerDied","Data":"edbd416b5f549490b0f73705dd05eb729fae7334b70cc7b1e075e9ec063a30d8"}
Nov 02 12:30:06.739441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.739425    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r"
Nov 02 12:30:06.739441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.739429    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwlg4r" event={"ID":"b106336e-aee7-4e00-bd14-0cd9ddf5da90","Type":"ContainerDied","Data":"93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305"}
Nov 02 12:30:06.739568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.739463    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ab2146a557d8568cba99d9fe67a8cdbdcc1d2617b3521646347760674f6305"
Nov 02 12:30:06.741480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.741449    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv" event={"ID":"cc6d77be-ea00-4c51-9da1-d2ad18cd97e0","Type":"ContainerDied","Data":"b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037"}
Nov 02 12:30:06.741598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.741478    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cr8khv"
Nov 02 12:30:06.741598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.741497    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2e032edf1db373947446a67f0aae08aa5104479c1167e489a3de76e62379037"
Nov 02 12:30:06.743815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.743790    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b" event={"ID":"a97f9058-0a18-495c-a764-2bacf1c207b3","Type":"ContainerDied","Data":"b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836"}
Nov 02 12:30:06.743815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.743806    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6j2b"
Nov 02 12:30:06.744006 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.743846    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b82a2058eae2bf85475c6cdce6d285c51874dcb9ff99513cb37e846b59f5e836"
Nov 02 12:30:06.745842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.745798    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr" event={"ID":"1cbdb3b2-47e8-42a3-8fed-f6199908778f","Type":"ContainerDied","Data":"2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c"}
Nov 02 12:30:06.745933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.745861    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae0a574b65ceed97b66f123f60667a411588080bef1f15c909d86324613e23c"
Nov 02 12:30:06.745933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.745808    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75nstr"
Nov 02 12:30:06.877054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:06.877026    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:30:07.018624 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.018573    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\") pod \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") "
Nov 02 12:30:07.018624 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.018627    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\") pod \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") "
Nov 02 12:30:07.018900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.018679    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\" (UniqueName: \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\") pod \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\" (UID: \"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd\") "
Nov 02 12:30:07.019425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.019399    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle" pod "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" (UID: "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:07.020788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.020765    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls" pod "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" (UID: "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd"). InnerVolumeSpecName "kube-api-access-hd4ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:07.023718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.023690    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util" pod "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" (UID: "7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:07.119378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.119298    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:07.119378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.119327    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:07.119378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.119339    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd4ls\" (UniqueName: \"kubernetes.io/projected/7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd-kube-api-access-hd4ls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:07.752552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.752516    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh"
Nov 02 12:30:07.752552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.752542    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq8jbh" event={"ID":"7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd","Type":"ContainerDied","Data":"03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8"}
Nov 02 12:30:07.753016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.752590    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03054c7709028a4c2790e5f4c923a9c16e34034fb4af8f12eb404867d1028ec8"
Nov 02 12:30:07.876981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:07.876955    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:30:08.028577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.028492    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\" (UniqueName: \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\") pod \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") "
Nov 02 12:30:08.028749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.028595    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\") pod \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") "
Nov 02 12:30:08.028749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.028638    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\") pod \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\" (UID: \"87be9f36-afe8-4b0a-9e9e-55489a558bbd\") "
Nov 02 12:30:08.029177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.029152    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle" pod "87be9f36-afe8-4b0a-9e9e-55489a558bbd" (UID: "87be9f36-afe8-4b0a-9e9e-55489a558bbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:08.030682 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.030654    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp" pod "87be9f36-afe8-4b0a-9e9e-55489a558bbd" (UID: "87be9f36-afe8-4b0a-9e9e-55489a558bbd"). InnerVolumeSpecName "kube-api-access-vjgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:08.034164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.034141    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util" pod "87be9f36-afe8-4b0a-9e9e-55489a558bbd" (UID: "87be9f36-afe8-4b0a-9e9e-55489a558bbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:08.129876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.129841    4928 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-util\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:08.129876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.129870    4928 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87be9f36-afe8-4b0a-9e9e-55489a558bbd-bundle\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:08.129876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.129880    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjgsp\" (UniqueName: \"kubernetes.io/projected/87be9f36-afe8-4b0a-9e9e-55489a558bbd-kube-api-access-vjgsp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:08.758939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.758857    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2" event={"ID":"87be9f36-afe8-4b0a-9e9e-55489a558bbd","Type":"ContainerDied","Data":"a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9"}
Nov 02 12:30:08.758939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.758887    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mrxw2"
Nov 02 12:30:08.758939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:08.758914    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a79a98fee2bb73baa322e093cf77216b3782bdfe16450f9bf53f7a3dc2b149f9"
Nov 02 12:30:09.727218 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:30:09.727178    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:30:11.018325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018290    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"]
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018592    4928 cpu_manager.go:414] "containerMap: removing container" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018603    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018614    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018619    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018627    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018634    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018642    4928 cpu_manager.go:414] "containerMap: removing container" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018646    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018654    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018658    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018666    4928 cpu_manager.go:414] "containerMap: removing container" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018670    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018676    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018680    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018686    4928 cpu_manager.go:414] "containerMap: removing container" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018690    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="extract"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018700    4928 cpu_manager.go:414] "containerMap: removing container" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018704    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018711    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018716    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018724    4928 cpu_manager.go:414] "containerMap: removing container" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018728    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018734    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018738    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="pull"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018749    4928 cpu_manager.go:414] "containerMap: removing container" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018754    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="util"
Nov 02 12:30:11.018746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018760    4928 cpu_manager.go:414] "containerMap: removing container" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018764    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018772    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018776    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018782    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018786    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018792    4928 cpu_manager.go:414] "containerMap: removing container" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018796    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018802    4928 cpu_manager.go:414] "containerMap: removing container" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018806    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018817    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018821    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018841    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018845    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018857    4928 cpu_manager.go:414] "containerMap: removing container" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018864    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018874    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018879    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="util"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018884    4928 cpu_manager.go:414] "containerMap: removing container" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018888    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="pull"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018893    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018898    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018944    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bec8dc-7794-489b-ab4e-64455c2f5a70" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018950    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac81d82-7f9e-42b2-9cdd-c2c33d3ee6bd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018955    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="87be9f36-afe8-4b0a-9e9e-55489a558bbd" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018961    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97f9058-0a18-495c-a764-2bacf1c207b3" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018968    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbdb3b2-47e8-42a3-8fed-f6199908778f" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018973    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b106336e-aee7-4e00-bd14-0cd9ddf5da90" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018978    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="423dc021-9142-468f-98bc-174b8a50e096" containerName="extract"
Nov 02 12:30:11.019622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.018983    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6d77be-ea00-4c51-9da1-d2ad18cd97e0" containerName="extract"
Nov 02 12:30:11.022302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.022285    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.037316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.037289    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"]
Nov 02 12:30:11.049188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.049154    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:30:11.049341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.049161    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 12:30:11.049341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.049273    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wt9d8\""
Nov 02 12:30:11.053639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.053612    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77737d7-f235-4e77-bdb7-d5e832779c03-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.053801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.053670    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhl7h\" (UniqueName: \"kubernetes.io/projected/f77737d7-f235-4e77-bdb7-d5e832779c03-kube-api-access-lhl7h\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.154154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.154102    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhl7h\" (UniqueName: \"kubernetes.io/projected/f77737d7-f235-4e77-bdb7-d5e832779c03-kube-api-access-lhl7h\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.154368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.154195    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77737d7-f235-4e77-bdb7-d5e832779c03-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.154616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.154592    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77737d7-f235-4e77-bdb7-d5e832779c03-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.179001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.178972    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhl7h\" (UniqueName: \"kubernetes.io/projected/f77737d7-f235-4e77-bdb7-d5e832779c03-kube-api-access-lhl7h\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg\" (UID: \"f77737d7-f235-4e77-bdb7-d5e832779c03\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.334109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.334021    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"
Nov 02 12:30:11.662646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.662620    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg"]
Nov 02 12:30:11.665535 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:11.665507    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77737d7_f235_4e77_bdb7_d5e832779c03.slice/crio-0f11f4feee33c5f0bcfb50927e74ba7717d7445528426d63bc518486a0def4ba WatchSource:0}: Error finding container 0f11f4feee33c5f0bcfb50927e74ba7717d7445528426d63bc518486a0def4ba: Status 404 returned error can't find the container with id 0f11f4feee33c5f0bcfb50927e74ba7717d7445528426d63bc518486a0def4ba
Nov 02 12:30:11.775299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:11.775263    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg" event={"ID":"f77737d7-f235-4e77-bdb7-d5e832779c03","Type":"ContainerStarted","Data":"0f11f4feee33c5f0bcfb50927e74ba7717d7445528426d63bc518486a0def4ba"}
Nov 02 12:30:13.786799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:13.786762    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg" event={"ID":"f77737d7-f235-4e77-bdb7-d5e832779c03","Type":"ContainerStarted","Data":"778566757d701ba076e50ec50a4c9810afd57b876b357e26a535c8733f545efb"}
Nov 02 12:30:13.860444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:13.860367    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xm6bg" podStartSLOduration=2.565212719 podStartE2EDuration="3.860348172s" totalImagesPullingTime="1.295135453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:13.846261494 +0000 UTC m=+211.495315451" watchObservedRunningTime="2026-11-02 12:30:13.860348172 +0000 UTC m=+211.509402130"
Nov 02 12:30:23.248541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.248503    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"]
Nov 02 12:30:23.252037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.252017    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.272160 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.272121    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"]
Nov 02 12:30:23.284126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.284085    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 12:30:23.284309 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.284127    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 12:30:23.284309 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.284166    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sg5ct\""
Nov 02 12:30:23.352430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.352397    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.352611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.352446    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pp56\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-kube-api-access-5pp56\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.453879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.453818    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.454058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.453908    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pp56\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-kube-api-access-5pp56\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.485551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.485507    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pp56\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-kube-api-access-5pp56\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.486002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.485979    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd8d875-0aab-429e-bc73-73f4b81ff401-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nwggq\" (UID: \"2fd8d875-0aab-429e-bc73-73f4b81ff401\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.580324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.580246    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"
Nov 02 12:30:23.882644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.882540    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"]
Nov 02 12:30:23.886110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.886089    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq"]
Nov 02 12:30:23.886203 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.886177    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.888482 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:23.888460    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fd8d875_0aab_429e_bc73_73f4b81ff401.slice/crio-3ab24462ccf22b0baebc9d5cdf26dda1bdf4916ce8a613d5d63458268501ac4a WatchSource:0}: Error finding container 3ab24462ccf22b0baebc9d5cdf26dda1bdf4916ce8a613d5d63458268501ac4a: Status 404 returned error can't find the container with id 3ab24462ccf22b0baebc9d5cdf26dda1bdf4916ce8a613d5d63458268501ac4a
Nov 02 12:30:23.903029 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.903009    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 12:30:23.906663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.906637    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 12:30:23.907045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.906773    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 12:30:23.907045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.906809    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwbqf\""
Nov 02 12:30:23.907247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.907232    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:23.907306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.907257    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:23.936108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.936083    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"]
Nov 02 12:30:23.958811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.958784    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.958930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.958834    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwddl\" (UniqueName: \"kubernetes.io/projected/201c8528-c25d-422c-a5ab-b2ad790f6a65-kube-api-access-dwddl\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.959001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.958981    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.959036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.959011    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.959094 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.959081    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:23.959148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:23.959130    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.059929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.059893    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.059929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.059937    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.060213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.059961    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwddl\" (UniqueName: \"kubernetes.io/projected/201c8528-c25d-422c-a5ab-b2ad790f6a65-kube-api-access-dwddl\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.060213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.059999    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.060213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.060014    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.060213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.060045    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.060465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.060440    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.062391 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.062368    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/201c8528-c25d-422c-a5ab-b2ad790f6a65-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.062510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.062481    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.062877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.062856    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.062973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.062872    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201c8528-c25d-422c-a5ab-b2ad790f6a65-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.103368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.103337    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwddl\" (UniqueName: \"kubernetes.io/projected/201c8528-c25d-422c-a5ab-b2ad790f6a65-kube-api-access-dwddl\") pod \"opendatahub-operator-controller-manager-67b54f6f58-qc44f\" (UID: \"201c8528-c25d-422c-a5ab-b2ad790f6a65\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.209667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.209588    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:24.570325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.570292    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"]
Nov 02 12:30:24.571530 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:24.571493    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201c8528_c25d_422c_a5ab_b2ad790f6a65.slice/crio-dbf139779a3515440258154af1ad584f0526c4a892c273812e59e5ec8d6fd722 WatchSource:0}: Error finding container dbf139779a3515440258154af1ad584f0526c4a892c273812e59e5ec8d6fd722: Status 404 returned error can't find the container with id dbf139779a3515440258154af1ad584f0526c4a892c273812e59e5ec8d6fd722
Nov 02 12:30:24.837774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.837680    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq" event={"ID":"2fd8d875-0aab-429e-bc73-73f4b81ff401","Type":"ContainerStarted","Data":"3ab24462ccf22b0baebc9d5cdf26dda1bdf4916ce8a613d5d63458268501ac4a"}
Nov 02 12:30:24.839328 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.839297    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f" event={"ID":"201c8528-c25d-422c-a5ab-b2ad790f6a65","Type":"ContainerStarted","Data":"dbf139779a3515440258154af1ad584f0526c4a892c273812e59e5ec8d6fd722"}
Nov 02 12:30:24.909840 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.909786    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:24.913668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.913639    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:24.930683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.930649    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:24.930683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.930660    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:24.930909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.930855    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nt9q8\""
Nov 02 12:30:24.936994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.936967    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:24.972269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.972229    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:24.972419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:24.972293    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhzf\" (UniqueName: \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.073353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.073316    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhzf\" (UniqueName: \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.073539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.073424    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.074047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.074016    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.098867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.098783    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhzf\" (UniqueName: \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k5ks5\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.229456 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.229409    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:25.564042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.564013    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:25.566995 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:25.566965    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036d2ba_b63d_42ba_946b_0b3276e55411.slice/crio-38d417f74971c061e66ae47d20470eb38532588f78fa3151c6d4b135973afae3 WatchSource:0}: Error finding container 38d417f74971c061e66ae47d20470eb38532588f78fa3151c6d4b135973afae3: Status 404 returned error can't find the container with id 38d417f74971c061e66ae47d20470eb38532588f78fa3151c6d4b135973afae3
Nov 02 12:30:25.848913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:25.848803    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" event={"ID":"4036d2ba-b63d-42ba-946b-0b3276e55411","Type":"ContainerStarted","Data":"38d417f74971c061e66ae47d20470eb38532588f78fa3151c6d4b135973afae3"}
Nov 02 12:30:26.864434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:26.864389    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq" event={"ID":"2fd8d875-0aab-429e-bc73-73f4b81ff401","Type":"ContainerStarted","Data":"845dcc2d1a5efe3031e20ee945ab3edcd9a0d3017b70d6595b53610cda2e2074"}
Nov 02 12:30:26.916443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:26.916385    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nwggq" podStartSLOduration=1.05305051 podStartE2EDuration="3.916370677s" totalImagesPullingTime="2.863320167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:26.914249556 +0000 UTC m=+224.563303513" watchObservedRunningTime="2026-11-02 12:30:26.916370677 +0000 UTC m=+224.565424635"
Nov 02 12:30:28.807962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.806494    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"]
Nov 02 12:30:28.810219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.810193    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:28.841861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.841817    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9crhr\""
Nov 02 12:30:28.844078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.844046    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"]
Nov 02 12:30:28.878182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.878135    4928 generic.go:356] "Generic (PLEG): container finished" podID="201c8528-c25d-422c-a5ab-b2ad790f6a65" containerID="b53fd1f68a9e752d4f919074f88bf36700e33cb6deaaecd158607e3e4f8dc6d1" exitCode=0
Nov 02 12:30:28.878342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.878283    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f" event={"ID":"201c8528-c25d-422c-a5ab-b2ad790f6a65","Type":"ContainerDied","Data":"b53fd1f68a9e752d4f919074f88bf36700e33cb6deaaecd158607e3e4f8dc6d1"}
Nov 02 12:30:28.911842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:28.911786    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dks5z\" (UniqueName: \"kubernetes.io/projected/675b6f92-53c6-4559-9391-218bfe57c01f-kube-api-access-dks5z\") pod \"limitador-operator-controller-manager-5778bb4449-v9xwk\" (UID: \"675b6f92-53c6-4559-9391-218bfe57c01f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:29.013144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.013053    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dks5z\" (UniqueName: \"kubernetes.io/projected/675b6f92-53c6-4559-9391-218bfe57c01f-kube-api-access-dks5z\") pod \"limitador-operator-controller-manager-5778bb4449-v9xwk\" (UID: \"675b6f92-53c6-4559-9391-218bfe57c01f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:29.041494 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.041464    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dks5z\" (UniqueName: \"kubernetes.io/projected/675b6f92-53c6-4559-9391-218bfe57c01f-kube-api-access-dks5z\") pod \"limitador-operator-controller-manager-5778bb4449-v9xwk\" (UID: \"675b6f92-53c6-4559-9391-218bfe57c01f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:29.127817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.127530    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:29.450625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.450593    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"]
Nov 02 12:30:29.454984 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:29.454944    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b6f92_53c6_4559_9391_218bfe57c01f.slice/crio-127dc7b7fc3112534dcc5b3001f32a5bef1b8004d3dd88fa77f19aaef3f54c78 WatchSource:0}: Error finding container 127dc7b7fc3112534dcc5b3001f32a5bef1b8004d3dd88fa77f19aaef3f54c78: Status 404 returned error can't find the container with id 127dc7b7fc3112534dcc5b3001f32a5bef1b8004d3dd88fa77f19aaef3f54c78
Nov 02 12:30:29.888097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.887474    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f" event={"ID":"201c8528-c25d-422c-a5ab-b2ad790f6a65","Type":"ContainerStarted","Data":"184114e19c6784d6fa90fe608cbf1d1c15f6389c4bb3bd3b0a77707eaeee5dae"}
Nov 02 12:30:29.888097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.887924    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:29.889606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.889543    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk" event={"ID":"675b6f92-53c6-4559-9391-218bfe57c01f","Type":"ContainerStarted","Data":"127dc7b7fc3112534dcc5b3001f32a5bef1b8004d3dd88fa77f19aaef3f54c78"}
Nov 02 12:30:29.969956 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:29.969896    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f" podStartSLOduration=2.51949375 podStartE2EDuration="6.969873451s" totalImagesPullingTime="3.938523155s" totalInitContainerRuntime="511.856546ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:29.967206496 +0000 UTC m=+227.616260491" watchObservedRunningTime="2026-11-02 12:30:29.969873451 +0000 UTC m=+227.618927410"
Nov 02 12:30:31.905169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:31.905122    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk" event={"ID":"675b6f92-53c6-4559-9391-218bfe57c01f","Type":"ContainerStarted","Data":"e4b8804c15ccf650381ff991dbe2eb6c7ed9d4c832e80ce3658967f62ccd29a3"}
Nov 02 12:30:31.905629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:31.905331    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:31.950226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:31.950152    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk" podStartSLOduration=2.020691296 podStartE2EDuration="3.950134782s" totalImagesPullingTime="1.929443486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:31.949284733 +0000 UTC m=+229.598338690" watchObservedRunningTime="2026-11-02 12:30:31.950134782 +0000 UTC m=+229.599188741"
Nov 02 12:30:32.773976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.773938    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"]
Nov 02 12:30:32.803676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.803642    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"]
Nov 02 12:30:32.803911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.803770    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:32.833102 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.833069    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j7f8s\""
Nov 02 12:30:32.849313 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.849280    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wzf\" (UniqueName: \"kubernetes.io/projected/fbd6770d-95d5-48fb-8fd2-c17cf1672c78-kube-api-access-b8wzf\") pod \"authorino-operator-7b7bb4bc77-b88v5\" (UID: \"fbd6770d-95d5-48fb-8fd2-c17cf1672c78\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:32.912462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.912427    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" event={"ID":"4036d2ba-b63d-42ba-946b-0b3276e55411","Type":"ContainerStarted","Data":"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"}
Nov 02 12:30:32.912810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.912524    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:32.949701 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.949669    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wzf\" (UniqueName: \"kubernetes.io/projected/fbd6770d-95d5-48fb-8fd2-c17cf1672c78-kube-api-access-b8wzf\") pod \"authorino-operator-7b7bb4bc77-b88v5\" (UID: \"fbd6770d-95d5-48fb-8fd2-c17cf1672c78\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:32.958765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.958716    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" podStartSLOduration=2.499971874 podStartE2EDuration="8.958701198s" totalImagesPullingTime="6.458729324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:32.957389137 +0000 UTC m=+230.606443106" watchObservedRunningTime="2026-11-02 12:30:32.958701198 +0000 UTC m=+230.607755155"
Nov 02 12:30:32.974186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:32.974159    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wzf\" (UniqueName: \"kubernetes.io/projected/fbd6770d-95d5-48fb-8fd2-c17cf1672c78-kube-api-access-b8wzf\") pod \"authorino-operator-7b7bb4bc77-b88v5\" (UID: \"fbd6770d-95d5-48fb-8fd2-c17cf1672c78\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:33.133993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:33.133912    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:33.463068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:33.463041    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"]
Nov 02 12:30:33.465611 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:33.465576    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd6770d_95d5_48fb_8fd2_c17cf1672c78.slice/crio-1191fb19e8f17ba58e6d7e83e5fabe306af6aa4f586d0ab4dcaafde1e0dc9661 WatchSource:0}: Error finding container 1191fb19e8f17ba58e6d7e83e5fabe306af6aa4f586d0ab4dcaafde1e0dc9661: Status 404 returned error can't find the container with id 1191fb19e8f17ba58e6d7e83e5fabe306af6aa4f586d0ab4dcaafde1e0dc9661
Nov 02 12:30:33.918851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:33.918798    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5" event={"ID":"fbd6770d-95d5-48fb-8fd2-c17cf1672c78","Type":"ContainerStarted","Data":"1191fb19e8f17ba58e6d7e83e5fabe306af6aa4f586d0ab4dcaafde1e0dc9661"}
Nov 02 12:30:34.926332 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:34.926282    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5" event={"ID":"fbd6770d-95d5-48fb-8fd2-c17cf1672c78","Type":"ContainerStarted","Data":"393f9e99f418af536bf27a73b23adae824ca93990e9ecad6042d2b3ac33143c5"}
Nov 02 12:30:34.926767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:34.926378    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:34.973358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:34.973291    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5" podStartSLOduration=1.676796073 podStartE2EDuration="2.973269201s" totalImagesPullingTime="1.296473128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:34.971983844 +0000 UTC m=+232.621037800" watchObservedRunningTime="2026-11-02 12:30:34.973269201 +0000 UTC m=+232.622323163"
Nov 02 12:30:35.431213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.431111    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"]
Nov 02 12:30:35.434776 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.434752    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:35.454121 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.454097    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9mh7b\""
Nov 02 12:30:35.468533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.468508    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 12:30:35.469550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.469524    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"]
Nov 02 12:30:35.572977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.572931    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngsv\" (UniqueName: \"kubernetes.io/projected/69b89c26-ba10-4856-8e9c-d13589d2ac73-kube-api-access-cngsv\") pod \"dns-operator-controller-manager-8557cbdb84-cxwbd\" (UID: \"69b89c26-ba10-4856-8e9c-d13589d2ac73\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:35.674333 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.674296    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngsv\" (UniqueName: \"kubernetes.io/projected/69b89c26-ba10-4856-8e9c-d13589d2ac73-kube-api-access-cngsv\") pod \"dns-operator-controller-manager-8557cbdb84-cxwbd\" (UID: \"69b89c26-ba10-4856-8e9c-d13589d2ac73\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:35.699073 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.699007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngsv\" (UniqueName: \"kubernetes.io/projected/69b89c26-ba10-4856-8e9c-d13589d2ac73-kube-api-access-cngsv\") pod \"dns-operator-controller-manager-8557cbdb84-cxwbd\" (UID: \"69b89c26-ba10-4856-8e9c-d13589d2ac73\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:35.746945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:35.746909    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:36.047336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:36.047302    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"]
Nov 02 12:30:36.049399 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:36.049366    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b89c26_ba10_4856_8e9c_d13589d2ac73.slice/crio-f54e3116fec3745446eca3bc85dff72c3ad0804a456bd4a19859539bbd922596 WatchSource:0}: Error finding container f54e3116fec3745446eca3bc85dff72c3ad0804a456bd4a19859539bbd922596: Status 404 returned error can't find the container with id f54e3116fec3745446eca3bc85dff72c3ad0804a456bd4a19859539bbd922596
Nov 02 12:30:36.937607 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:36.937571    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd" event={"ID":"69b89c26-ba10-4856-8e9c-d13589d2ac73","Type":"ContainerStarted","Data":"f54e3116fec3745446eca3bc85dff72c3ad0804a456bd4a19859539bbd922596"}
Nov 02 12:30:37.944085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:37.944050    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd" event={"ID":"69b89c26-ba10-4856-8e9c-d13589d2ac73","Type":"ContainerStarted","Data":"5f3c5c3a7e68aa4e6350a6158106190988a98154eddbfd2d5aa7087dd7835a52"}
Nov 02 12:30:37.944537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:37.944198    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:37.988546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:37.988496    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd" podStartSLOduration=1.235821988 podStartE2EDuration="2.988482334s" totalImagesPullingTime="1.752660346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:37.987609516 +0000 UTC m=+235.636663476" watchObservedRunningTime="2026-11-02 12:30:37.988482334 +0000 UTC m=+235.637536291"
Nov 02 12:30:39.727575 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:30:39.727537    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:30:40.899495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:40.899453    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-qc44f"
Nov 02 12:30:42.915685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:42.915652    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-v9xwk"
Nov 02 12:30:43.921099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:43.921069    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:44.440955 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.440915    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 12:30:44.446101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.446076    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.469588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.469556    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 12:30:44.556836 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.556799    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjl5t\" (UniqueName: \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.557015 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.556867    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.658216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.658176    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.658406 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.658291    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjl5t\" (UniqueName: \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.658612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.658586    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.691677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.691619    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjl5t\" (UniqueName: \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\") pod \"kuadrant-operator-controller-manager-678d87766f-jpm5x\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:44.759465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:44.759430    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:45.094783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:45.094629    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 12:30:45.098076 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:45.098047    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a43c64c_0e99_4fb3_a131_042645c331f8.slice/crio-e3ba84f0d858affb675b710654b0d5ba49161bb8e7ff5a313feb58c795b78cb0 WatchSource:0}: Error finding container e3ba84f0d858affb675b710654b0d5ba49161bb8e7ff5a313feb58c795b78cb0: Status 404 returned error can't find the container with id e3ba84f0d858affb675b710654b0d5ba49161bb8e7ff5a313feb58c795b78cb0
Nov 02 12:30:45.933531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:45.933498    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b88v5"
Nov 02 12:30:45.994318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:45.994285    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" event={"ID":"3a43c64c-0e99-4fb3-a131-042645c331f8","Type":"ContainerStarted","Data":"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"}
Nov 02 12:30:45.994502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:45.994337    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" event={"ID":"3a43c64c-0e99-4fb3-a131-042645c331f8","Type":"ContainerStarted","Data":"e3ba84f0d858affb675b710654b0d5ba49161bb8e7ff5a313feb58c795b78cb0"}
Nov 02 12:30:45.994502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:45.994403    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:46.044945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:46.044884    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" podStartSLOduration=2.044861804 podStartE2EDuration="2.044861804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:46.04351624 +0000 UTC m=+243.692570199" watchObservedRunningTime="2026-11-02 12:30:46.044861804 +0000 UTC m=+243.693915762"
Nov 02 12:30:48.951294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:48.951263    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cxwbd"
Nov 02 12:30:53.756269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.756233    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"]
Nov 02 12:30:53.761056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.761024    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.782591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.782557    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-68v8w\""
Nov 02 12:30:53.785376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.785349    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"]
Nov 02 12:30:53.940601 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940549    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.940601 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940599    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.940878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940663    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4md\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-kube-api-access-hq4md\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.940878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940753    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bee30f-e272-4cf6-8b8c-983855b3107c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.940878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940799    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.940878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940833    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.941155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940936    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.941155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.940978    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:53.941155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:53.941054    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042455    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042488    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042519    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042540    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042568    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042618    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042637    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042655    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4md\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-kube-api-access-hq4md\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.042784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.042686    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bee30f-e272-4cf6-8b8c-983855b3107c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.043109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.043061    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.043184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.043116    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.043245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.043174    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.043305 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.043269    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.043430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.043408    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bee30f-e272-4cf6-8b8c-983855b3107c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.045030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.045007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.045173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.045155    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.078263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.078238    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.078447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.078428    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4md\" (UniqueName: \"kubernetes.io/projected/69bee30f-e272-4cf6-8b8c-983855b3107c-kube-api-access-hq4md\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fnlqcc\" (UID: \"69bee30f-e272-4cf6-8b8c-983855b3107c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.375875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.375793    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:54.695858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.695812    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"]
Nov 02 12:30:54.698343 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:30:54.698309    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bee30f_e272_4cf6_8b8c_983855b3107c.slice/crio-33e506b8654b2758c981179d0c2ac1d345802068edec16656465e15cc220bcb3 WatchSource:0}: Error finding container 33e506b8654b2758c981179d0c2ac1d345802068edec16656465e15cc220bcb3: Status 404 returned error can't find the container with id 33e506b8654b2758c981179d0c2ac1d345802068edec16656465e15cc220bcb3
Nov 02 12:30:54.700630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.700592    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:30:54.700730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.700677    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:30:54.700730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:54.700722    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:30:55.047565 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:55.047527    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc" event={"ID":"69bee30f-e272-4cf6-8b8c-983855b3107c","Type":"ContainerStarted","Data":"392752b1634c3cacf39494b963fa325fd5cc8a2c1b67a45747eb649ad3f77e7f"}
Nov 02 12:30:55.048027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:55.047585    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc" event={"ID":"69bee30f-e272-4cf6-8b8c-983855b3107c","Type":"ContainerStarted","Data":"33e506b8654b2758c981179d0c2ac1d345802068edec16656465e15cc220bcb3"}
Nov 02 12:30:55.096649 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:55.096599    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc" podStartSLOduration=2.096583838 podStartE2EDuration="2.096583838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:55.095496762 +0000 UTC m=+252.744550725" watchObservedRunningTime="2026-11-02 12:30:55.096583838 +0000 UTC m=+252.745637795"
Nov 02 12:30:55.376306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:55.376215    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:55.381069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:55.381041    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:56.053292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:56.053255    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:56.054270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:56.054247    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fnlqcc"
Nov 02 12:30:57.001923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.001889    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 12:30:57.095569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.095529    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:57.096127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.095840    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" podUID="4036d2ba-b63d-42ba-946b-0b3276e55411" containerName="manager" containerID="cri-o://9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010" gracePeriod=10
Nov 02 12:30:57.442280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.442251    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:57.468837 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.468806    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\" (UniqueName: \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\") pod \"4036d2ba-b63d-42ba-946b-0b3276e55411\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") "
Nov 02 12:30:57.468961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.468911    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\") pod \"4036d2ba-b63d-42ba-946b-0b3276e55411\" (UID: \"4036d2ba-b63d-42ba-946b-0b3276e55411\") "
Nov 02 12:30:57.469298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.469273    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume" pod "4036d2ba-b63d-42ba-946b-0b3276e55411" (UID: "4036d2ba-b63d-42ba-946b-0b3276e55411"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:57.471005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.470981    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf" pod "4036d2ba-b63d-42ba-946b-0b3276e55411" (UID: "4036d2ba-b63d-42ba-946b-0b3276e55411"). InnerVolumeSpecName "kube-api-access-bdhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:57.569572 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.569483    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhzf\" (UniqueName: \"kubernetes.io/projected/4036d2ba-b63d-42ba-946b-0b3276e55411-kube-api-access-bdhzf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:57.569572 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:57.569516    4928 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4036d2ba-b63d-42ba-946b-0b3276e55411-extensions-socket-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:30:58.063301 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.063265    4928 generic.go:356] "Generic (PLEG): container finished" podID="4036d2ba-b63d-42ba-946b-0b3276e55411" containerID="9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010" exitCode=0
Nov 02 12:30:58.063465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.063317    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"
Nov 02 12:30:58.063465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.063353    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" event={"ID":"4036d2ba-b63d-42ba-946b-0b3276e55411","Type":"ContainerDied","Data":"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"}
Nov 02 12:30:58.063465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.063408    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5" event={"ID":"4036d2ba-b63d-42ba-946b-0b3276e55411","Type":"ContainerDied","Data":"38d417f74971c061e66ae47d20470eb38532588f78fa3151c6d4b135973afae3"}
Nov 02 12:30:58.063465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.063439    4928 scope.go:122] "RemoveContainer" containerID="9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"
Nov 02 12:30:58.072860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.072837    4928 scope.go:122] "RemoveContainer" containerID="9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"
Nov 02 12:30:58.073136 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:30:58.073108    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010\": container with ID starting with 9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010 not found: ID does not exist" containerID="9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"
Nov 02 12:30:58.073183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.073150    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010"} err="failed to get container status \"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010\": rpc error: code = NotFound desc = could not find container \"9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010\": container with ID starting with 9626bd33dfec36a010ffbf3e7930f4541a214229463d04a02edb61016322b010 not found: ID does not exist"
Nov 02 12:30:58.144998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.144963    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:58.158680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.158645    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k5ks5"]
Nov 02 12:30:58.496757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:30:58.496721    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4036d2ba-b63d-42ba-946b-0b3276e55411" path="/var/lib/kubelet/pods/4036d2ba-b63d-42ba-946b-0b3276e55411/volumes"
Nov 02 12:31:01.258148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.258112    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"]
Nov 02 12:31:01.258530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.258516    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4036d2ba-b63d-42ba-946b-0b3276e55411" containerName="manager"
Nov 02 12:31:01.258574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.258532    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4036d2ba-b63d-42ba-946b-0b3276e55411" containerName="manager"
Nov 02 12:31:01.258636 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.258627    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4036d2ba-b63d-42ba-946b-0b3276e55411" containerName="manager"
Nov 02 12:31:01.301591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.301563    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"]
Nov 02 12:31:01.301735 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.301663    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.320764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.320736    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-md54s\""
Nov 02 12:31:01.334339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.334312    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 12:31:01.334339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.334328    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 12:31:01.334339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.334319    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 12:31:01.334531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.334344    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 12:31:01.334531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.334355    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:31:01.399228 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.399197    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-metrics-certs\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.399371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.399250    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c0ff80f-122a-416e-a090-6d0ac0b0de76-manager-config\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.399371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.399322    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-cert\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.399451 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.399425    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9h7\" (UniqueName: \"kubernetes.io/projected/5c0ff80f-122a-416e-a090-6d0ac0b0de76-kube-api-access-zv9h7\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.499851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.499788    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9h7\" (UniqueName: \"kubernetes.io/projected/5c0ff80f-122a-416e-a090-6d0ac0b0de76-kube-api-access-zv9h7\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.500022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.499945    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-metrics-certs\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.500022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.500006    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c0ff80f-122a-416e-a090-6d0ac0b0de76-manager-config\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.500145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.500049    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-cert\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.500703 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.500678    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c0ff80f-122a-416e-a090-6d0ac0b0de76-manager-config\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.502531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.502500    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-cert\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.502613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.502515    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0ff80f-122a-416e-a090-6d0ac0b0de76-metrics-certs\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.524225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.524176    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9h7\" (UniqueName: \"kubernetes.io/projected/5c0ff80f-122a-416e-a090-6d0ac0b0de76-kube-api-access-zv9h7\") pod \"jobset-controller-manager-6757d7bffd-f8cc7\" (UID: \"5c0ff80f-122a-416e-a090-6d0ac0b0de76\") " pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.630153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.630002    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:01.962436 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:01.962405    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"]
Nov 02 12:31:01.964204 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:31:01.964178    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0ff80f_122a_416e_a090_6d0ac0b0de76.slice/crio-14acaf49e2b9fcbc00d4e6923cb0f8c08e80d023d1903f733184035437af3982 WatchSource:0}: Error finding container 14acaf49e2b9fcbc00d4e6923cb0f8c08e80d023d1903f733184035437af3982: Status 404 returned error can't find the container with id 14acaf49e2b9fcbc00d4e6923cb0f8c08e80d023d1903f733184035437af3982
Nov 02 12:31:02.086987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:02.086942    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7" event={"ID":"5c0ff80f-122a-416e-a090-6d0ac0b0de76","Type":"ContainerStarted","Data":"14acaf49e2b9fcbc00d4e6923cb0f8c08e80d023d1903f733184035437af3982"}
Nov 02 12:31:06.111360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:06.111316    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7" event={"ID":"5c0ff80f-122a-416e-a090-6d0ac0b0de76","Type":"ContainerStarted","Data":"6a10a01d5235dfd8ffde2be2c2f37c7cf791e33443ae0f15bdd0d0810b30228b"}
Nov 02 12:31:06.111873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:06.111418    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:06.155257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:06.155206    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7" podStartSLOduration=2.104101285 podStartE2EDuration="5.155193379s" totalImagesPullingTime="3.051092094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:06.153730712 +0000 UTC m=+263.802784670" watchObservedRunningTime="2026-11-02 12:31:06.155193379 +0000 UTC m=+263.804247336"
Nov 02 12:31:09.728042 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:31:09.728001    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:31:17.122643 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:17.122609    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6757d7bffd-f8cc7"
Nov 02 12:31:39.728389 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:31:39.728345    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:31:42.524346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:42.524318    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:31:42.525558 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:31:42.525538    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:32:09.727710 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:32:09.727624    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:32:31.483055 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:31.483006    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:32:31.483576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:31.483114    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:32:33.591738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.591702    4928 generic.go:356] "Generic (PLEG): container finished" podID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerID="fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f" exitCode=0
Nov 02 12:32:33.592174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.591747    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerDied","Data":"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"}
Nov 02 12:32:33.592218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.592198    4928 scope.go:122] "RemoveContainer" containerID="fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"
Nov 02 12:32:33.593374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.593343    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:32:33.593426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.593413    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:32:33.593476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:33.593451    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:32:34.598152 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:34.598116    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerStarted","Data":"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"}
Nov 02 12:32:35.282127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:35.282089    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:32:35.282342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:35.282323    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 12:32:35.282392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:35.282370    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 12:32:36.281510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:36.281473    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 12:32:36.281944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:36.281528    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 12:32:37.281912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:37.281868    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 12:32:37.282313 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:37.281928    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 12:32:38.281578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:38.281541    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 12:32:38.281754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:38.281596    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 12:32:39.281691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:39.281650    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 12:32:39.282135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:39.281717    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 12:32:39.727153 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:32:39.727070    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:32:40.284733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:40.284699    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:32:40.285178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:40.285066    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:32:40.285775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:32:40.285756    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:33:09.727977 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:33:09.727938    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:33:39.728041 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:33:39.728001    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:34:09.730924 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:34:09.730879    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:34:10.119468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.119422    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"]
Nov 02 12:34:10.125906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.125847    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.147379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.147341    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"]
Nov 02 12:34:10.156859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.156792    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 12:34:10.157009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.156862    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 12:34:10.158368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.158343    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lwrqm\""
Nov 02 12:34:10.158368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.158345    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 12:34:10.242665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.242611    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q222w\" (UniqueName: \"kubernetes.io/projected/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-kube-api-access-q222w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.242903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.242684    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.242903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.242790    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.242903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.242866    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.243116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.242976    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.243116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.243045    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.343806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.343749    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.343838    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q222w\" (UniqueName: \"kubernetes.io/projected/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-kube-api-access-q222w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.343887    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.343916    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.343963    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.344026    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.344380    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.344484    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.344751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.344723    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.347275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.347248    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.347521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.347497    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.369044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.369015    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q222w\" (UniqueName: \"kubernetes.io/projected/e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7-kube-api-access-q222w\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hrm6f\" (UID: \"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.447261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.447125    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:10.759867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.759786    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"]
Nov 02 12:34:10.762193 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:34:10.762162    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fdc759_e3a5_4368_88d9_eb9f1e31a7a7.slice/crio-f39a85098fb7006573a2d0ef8505dcb1f3d687377d39e44e13df18190a8d2f91 WatchSource:0}: Error finding container f39a85098fb7006573a2d0ef8505dcb1f3d687377d39e44e13df18190a8d2f91: Status 404 returned error can't find the container with id f39a85098fb7006573a2d0ef8505dcb1f3d687377d39e44e13df18190a8d2f91
Nov 02 12:34:10.763571 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:10.763554    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:34:11.089749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:11.089648    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f" event={"ID":"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7","Type":"ContainerStarted","Data":"f39a85098fb7006573a2d0ef8505dcb1f3d687377d39e44e13df18190a8d2f91"}
Nov 02 12:34:12.096965 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:12.096926    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f" event={"ID":"e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7","Type":"ContainerStarted","Data":"212b9f72e24c3be5fd2775c572437ff25e5bf794af234f8782999a67e073a344"}
Nov 02 12:34:12.144357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:12.144302    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f" podStartSLOduration=0.952012917 podStartE2EDuration="2.144285855s" totalImagesPullingTime="1.192272938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:12.142363568 +0000 UTC m=+449.791417526" watchObservedRunningTime="2026-11-02 12:34:12.144285855 +0000 UTC m=+449.793339813"
Nov 02 12:34:12.447231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:12.447197    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:14.189165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.189128    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"]
Nov 02 12:34:14.192839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.192802    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.214021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.213647    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 12:34:14.229401 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.229373    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"]
Nov 02 12:34:14.282785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.282740    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.282960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.282797    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.282960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.282862    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.283056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.282960    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6lfd\" (UniqueName: \"kubernetes.io/projected/c51b659b-07c3-40f5-ad34-c8fc405145d5-kube-api-access-f6lfd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.283056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.283005    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b659b-07c3-40f5-ad34-c8fc405145d5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.283122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.283107    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384237 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384201    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384242    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384287    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lfd\" (UniqueName: \"kubernetes.io/projected/c51b659b-07c3-40f5-ad34-c8fc405145d5-kube-api-access-f6lfd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384319    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b659b-07c3-40f5-ad34-c8fc405145d5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384373    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384405    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384672    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384738    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.384854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.384782    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.386780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.386754    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b659b-07c3-40f5-ad34-c8fc405145d5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.387060 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.387041    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b659b-07c3-40f5-ad34-c8fc405145d5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.408843 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.408809    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lfd\" (UniqueName: \"kubernetes.io/projected/c51b659b-07c3-40f5-ad34-c8fc405145d5-kube-api-access-f6lfd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt\" (UID: \"c51b659b-07c3-40f5-ad34-c8fc405145d5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.521625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.521593    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:14.825146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:14.825119    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"]
Nov 02 12:34:15.113329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:15.113295    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt" event={"ID":"c51b659b-07c3-40f5-ad34-c8fc405145d5","Type":"ContainerStarted","Data":"abfeb4e5dfddad995a658894add3a09496132da2c427f0cc73ecf6448f812787"}
Nov 02 12:34:16.121134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:16.121098    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt" event={"ID":"c51b659b-07c3-40f5-ad34-c8fc405145d5","Type":"ContainerStarted","Data":"1267ca05d13e75db41d8e9d7d74d07094ecde76a362a340030efdbda43af037d"}
Nov 02 12:34:16.165623 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:16.165549    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt" podStartSLOduration=1.894538045 podStartE2EDuration="2.165530492s" totalImagesPullingTime="270.992447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:16.16469587 +0000 UTC m=+453.813749828" watchObservedRunningTime="2026-11-02 12:34:16.165530492 +0000 UTC m=+453.814584451"
Nov 02 12:34:16.522650 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:16.522612    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:20.448332 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.448290    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:20.460894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.460864    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:20.711987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.711907    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"]
Nov 02 12:34:20.716161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.716137    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.734354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.734320    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 12:34:20.737792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.737763    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"]
Nov 02 12:34:20.842112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842072    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.842316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842124    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.842316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842203    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptx6\" (UniqueName: \"kubernetes.io/projected/39a8e1f5-a620-4814-8913-5ed14af5ccdd-kube-api-access-fptx6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.842458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842316    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.842458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842343    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.842458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.842400    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943626    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943677    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943724    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943773    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943809    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.943913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.943896    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptx6\" (UniqueName: \"kubernetes.io/projected/39a8e1f5-a620-4814-8913-5ed14af5ccdd-kube-api-access-fptx6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.944220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.944150    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.944331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.944308    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.944445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.944424    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.946790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.946757    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a8e1f5-a620-4814-8913-5ed14af5ccdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.946923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.946905    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a8e1f5-a620-4814-8913-5ed14af5ccdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:20.968719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:20.968652    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptx6\" (UniqueName: \"kubernetes.io/projected/39a8e1f5-a620-4814-8913-5ed14af5ccdd-kube-api-access-fptx6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95\" (UID: \"39a8e1f5-a620-4814-8913-5ed14af5ccdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:21.032265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.032225    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:21.178630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.178599    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hrm6f"
Nov 02 12:34:21.353440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.353401    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"]
Nov 02 12:34:21.355271 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:34:21.355241    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a8e1f5_a620_4814_8913_5ed14af5ccdd.slice/crio-8c4cad8684b5219e55e86d2da1a82df14250eeb1f7d95ed31b0af62d5db43e04 WatchSource:0}: Error finding container 8c4cad8684b5219e55e86d2da1a82df14250eeb1f7d95ed31b0af62d5db43e04: Status 404 returned error can't find the container with id 8c4cad8684b5219e55e86d2da1a82df14250eeb1f7d95ed31b0af62d5db43e04
Nov 02 12:34:21.864444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.864138    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"]
Nov 02 12:34:21.868850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.868796    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.895182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.895078    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 12:34:21.895884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.895808    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"]
Nov 02 12:34:21.955005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.954973    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.955151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.955099    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpbt\" (UniqueName: \"kubernetes.io/projected/d3e5f98e-cab4-4a6b-8884-efda80d420ec-kube-api-access-jrpbt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.955193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.955155    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.955193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.955174    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.955281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.955197    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:21.955372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:21.955347    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056174    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056229    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056259    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056309    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056340    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.056493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.056429    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpbt\" (UniqueName: \"kubernetes.io/projected/d3e5f98e-cab4-4a6b-8884-efda80d420ec-kube-api-access-jrpbt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.058086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.058055    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.058262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.058080    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.058361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.058305    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.059689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.059667    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e5f98e-cab4-4a6b-8884-efda80d420ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.060450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.060426    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e5f98e-cab4-4a6b-8884-efda80d420ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.081385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.081355    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpbt\" (UniqueName: \"kubernetes.io/projected/d3e5f98e-cab4-4a6b-8884-efda80d420ec-kube-api-access-jrpbt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5\" (UID: \"d3e5f98e-cab4-4a6b-8884-efda80d420ec\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.162303 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.162201    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95" event={"ID":"39a8e1f5-a620-4814-8913-5ed14af5ccdd","Type":"ContainerStarted","Data":"0d7aafa3ee477ce37c23e66e4889610996276acb3440e7236470fdc732990a99"}
Nov 02 12:34:22.162303 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.162283    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95" event={"ID":"39a8e1f5-a620-4814-8913-5ed14af5ccdd","Type":"ContainerStarted","Data":"8c4cad8684b5219e55e86d2da1a82df14250eeb1f7d95ed31b0af62d5db43e04"}
Nov 02 12:34:22.198688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.198658    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:22.209452 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.209397    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95" podStartSLOduration=1.978530119 podStartE2EDuration="2.209383573s" totalImagesPullingTime="230.853454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:22.207067165 +0000 UTC m=+459.856121135" watchObservedRunningTime="2026-11-02 12:34:22.209383573 +0000 UTC m=+459.858437530"
Nov 02 12:34:22.505569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:22.505537    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"]
Nov 02 12:34:22.508919 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:34:22.508891    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3e5f98e_cab4_4a6b_8884_efda80d420ec.slice/crio-60374343cb0f79a5919b80ab58aa345595b46d8dbe007194c8205c3f31541028 WatchSource:0}: Error finding container 60374343cb0f79a5919b80ab58aa345595b46d8dbe007194c8205c3f31541028: Status 404 returned error can't find the container with id 60374343cb0f79a5919b80ab58aa345595b46d8dbe007194c8205c3f31541028
Nov 02 12:34:23.032726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:23.032636    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:23.176028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:23.175973    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" event={"ID":"d3e5f98e-cab4-4a6b-8884-efda80d420ec","Type":"ContainerStarted","Data":"bf5d4419ec4488492559984c5fde777f95efa4d3958d7be715fe6ea2c7974e70"}
Nov 02 12:34:23.176226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:23.176038    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" event={"ID":"d3e5f98e-cab4-4a6b-8884-efda80d420ec","Type":"ContainerStarted","Data":"60374343cb0f79a5919b80ab58aa345595b46d8dbe007194c8205c3f31541028"}
Nov 02 12:34:23.199457 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:23.199426    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:23.224793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:23.224735    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podStartSLOduration=1.997567919 podStartE2EDuration="2.224720663s" totalImagesPullingTime="227.152744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:23.223519192 +0000 UTC m=+460.872573152" watchObservedRunningTime="2026-11-02 12:34:23.224720663 +0000 UTC m=+460.873774620"
Nov 02 12:34:24.522358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:24.522318    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:24.535249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:24.535218    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:25.199566 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:25.199529    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt"
Nov 02 12:34:31.032662 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:31.032623    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:31.045252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:31.045226    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:31.234119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:31.234087    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95"
Nov 02 12:34:32.199846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:32.199802    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:32.212538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:32.212509    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:32.239802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:32.239770    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:34:35.846781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.846744    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"]
Nov 02 12:34:35.853284 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.853262    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.885172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.885145    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"]
Nov 02 12:34:35.980884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.980822    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.980911    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981011    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981047    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981149    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981177    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lp97\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-kube-api-access-5lp97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981265    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981289    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:35.981342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:35.981314    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082477    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082514    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082530    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082583    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082627    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082657    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082672    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082699    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.082755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.082714    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lp97\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-kube-api-access-5lp97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.083123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.083009    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.083162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.083138    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.083247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.083163    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.083320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.083289    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.083487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.083468    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.085075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.085054    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.085184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.085167    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.107326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.107263    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.107415 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.107338    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lp97\" (UniqueName: \"kubernetes.io/projected/ce79f7b4-4d9f-461c-bac8-5ff7576cf72f-kube-api-access-5lp97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4g2zz\" (UID: \"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.169324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.169293    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:36.493642 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:34:36.493614    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce79f7b4_4d9f_461c_bac8_5ff7576cf72f.slice/crio-3980cef82c59b3ceb4e95d1a9180fabd1cebc9f3eb06c3fc8f1676cf7f93e19e WatchSource:0}: Error finding container 3980cef82c59b3ceb4e95d1a9180fabd1cebc9f3eb06c3fc8f1676cf7f93e19e: Status 404 returned error can't find the container with id 3980cef82c59b3ceb4e95d1a9180fabd1cebc9f3eb06c3fc8f1676cf7f93e19e
Nov 02 12:34:36.495795 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.495761    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:36.495900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.495853    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:36.495900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.495887    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:36.497090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:36.497071    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"]
Nov 02 12:34:37.261357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:37.261313    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" event={"ID":"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f","Type":"ContainerStarted","Data":"e1c8532ae1c1781595e45d002681b13c3f9fe2182915de7135cff74793a2d260"}
Nov 02 12:34:37.261768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:37.261379    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" event={"ID":"ce79f7b4-4d9f-461c-bac8-5ff7576cf72f","Type":"ContainerStarted","Data":"3980cef82c59b3ceb4e95d1a9180fabd1cebc9f3eb06c3fc8f1676cf7f93e19e"}
Nov 02 12:34:37.324428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:37.324368    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podStartSLOduration=2.324350381 podStartE2EDuration="2.324350381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:37.323669128 +0000 UTC m=+474.972723085" watchObservedRunningTime="2026-11-02 12:34:37.324350381 +0000 UTC m=+474.973404339"
Nov 02 12:34:38.170056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:38.170025    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:38.171634 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:38.171601    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:38.171744 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:38.171661    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:39.170621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:39.170584    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:39.171055 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:39.170653    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:39.754203 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:34:39.754163    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:34:40.170398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:40.170312    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:40.170398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:40.170380    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:41.170331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:41.170288    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:41.170759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:41.170363    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:42.169856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:42.169794    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:42.170063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:42.169874    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:43.170222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:43.170172    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:34:43.170648 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:43.170252    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 02 12:34:45.171549 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:45.171481    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:45.172261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:45.172203    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:46.212961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:46.212929    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:46.213371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:46.213065    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:47.226548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:47.226503    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:47.227287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:47.227235    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:47.348224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:47.348191    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:47.348512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:47.348490    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:47.349388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:47.349366    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz"
Nov 02 12:34:50.893410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.893370    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:34:50.898395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.898369    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.925422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.925383    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:34:50.958836 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.958786    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.958853    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.958876    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959005    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwm5\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959044    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959062    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959081    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959248    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:50.959324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:50.959286    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060190    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwm5\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060243    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060261    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060276    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060309    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060324    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060363    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060402    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060571    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060808    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.060975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.060876    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.061049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.061006    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.061137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.061116    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.061247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.061226    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.062656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.062628    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.062904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.062884    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.084975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.084948    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.085218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.085197    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwm5\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zdk72\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.214064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.213976    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:51.534388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.534354    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:34:51.537531 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:34:51.537501    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993299a0_d111_4e9b_b2be_ae6e54b1a032.slice/crio-91ef55c063548e8a9f0e45a8e6d238d2729cbaa401f25d5953e8c693bfd32116 WatchSource:0}: Error finding container 91ef55c063548e8a9f0e45a8e6d238d2729cbaa401f25d5953e8c693bfd32116: Status 404 returned error can't find the container with id 91ef55c063548e8a9f0e45a8e6d238d2729cbaa401f25d5953e8c693bfd32116
Nov 02 12:34:51.539698 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.539660    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:51.539796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.539747    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:51.539880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:51.539796    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:34:52.363322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:52.363288    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" event={"ID":"993299a0-d111-4e9b-b2be-ae6e54b1a032","Type":"ContainerStarted","Data":"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"}
Nov 02 12:34:52.363717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:52.363341    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" event={"ID":"993299a0-d111-4e9b-b2be-ae6e54b1a032","Type":"ContainerStarted","Data":"91ef55c063548e8a9f0e45a8e6d238d2729cbaa401f25d5953e8c693bfd32116"}
Nov 02 12:34:52.451523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:52.450278    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podStartSLOduration=2.450256713 podStartE2EDuration="2.450256713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:52.423432043 +0000 UTC m=+490.072485998" watchObservedRunningTime="2026-11-02 12:34:52.450256713 +0000 UTC m=+490.099310672"
Nov 02 12:34:53.215103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:53.215065    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:53.216735 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:53.216709    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:53.216877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:53.216756    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:54.215108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:54.215022    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:54.215108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:54.215077    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:55.214591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:55.214555    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:55.214776 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:55.214608    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:56.214534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:56.214497    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:56.214951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:56.214565    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:57.214944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:57.214910    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:57.215352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:57.214964    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:58.214701 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:58.214667    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:34:58.214897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:58.214725    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 12:34:59.858978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:59.858930    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:59.859452 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:59.859337    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:34:59.860254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:34:59.860232    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:35:09.727418 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:35:09.727382    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:35:22.751705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.751668    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:35:22.757140 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.757114    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775377 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775342    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775482    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775603 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775580    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775637    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775750    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775792    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775817    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775881    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.775958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.775920    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.776220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.776201    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f080cb5f-openshift-default-dockercfg-hq7k6\""
Nov 02 12:35:22.792499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.792470    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:35:22.876804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.876770    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877086    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877148    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877188    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877215    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877238    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877264    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877345    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877304    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877490    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.877918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.877897    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.878490 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.878172    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.878762 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.878740    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.880724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.880629    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.880919 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.880888    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.882361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.882335    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.884291 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.884266    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:35:22.889360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.889343    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.902916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.902890    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.905673 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.905647    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\") pod \"e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") " pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:22.913768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.912199    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-53b6603d-openshift-default-dockercfg-zrbk2\""
Nov 02 12:35:22.914155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.914127    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:35:22.978512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978472    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978523    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978609    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978652    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978747    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978777    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncxd\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978796    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978865    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:22.978975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:22.978958    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.073625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.073538    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:23.079861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.079818    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.079985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.079872    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.079985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.079902    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.079985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.079926    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.079985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.079943    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080062    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080143    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncxd\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080200    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080350 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080237    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080532    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080755    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.080941 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080844    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.081018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.080813    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.081160 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.081138    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.082986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.082961    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.083319 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.083300    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.105690 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.105656    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.105889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.105763    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncxd\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\") pod \"e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") " pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.231488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.231453    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:23.406414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.406203    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:35:23.426338 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:23.426299    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46b7bd7_adeb_4624_8b6f_cf39a4b7ac5b.slice/crio-6fcb2db031a17e601e0084fe688e6b12f2040ee715b233b3d7eadd35496cc538 WatchSource:0}: Error finding container 6fcb2db031a17e601e0084fe688e6b12f2040ee715b233b3d7eadd35496cc538: Status 404 returned error can't find the container with id 6fcb2db031a17e601e0084fe688e6b12f2040ee715b233b3d7eadd35496cc538
Nov 02 12:35:23.428726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.428686    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.428890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.428766    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.428890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.428795    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.528392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.528350    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerStarted","Data":"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"}
Nov 02 12:35:23.528612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.528417    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerStarted","Data":"6fcb2db031a17e601e0084fe688e6b12f2040ee715b233b3d7eadd35496cc538"}
Nov 02 12:35:23.565226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.565194    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:35:23.565771 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:23.565744    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad91c1e_cc09_4114_a851_c5961a98bbc6.slice/crio-0a0ead4e384099d2650c4921134cca18e340982ee4f30c918f8de42fcf834bad WatchSource:0}: Error finding container 0a0ead4e384099d2650c4921134cca18e340982ee4f30c918f8de42fcf834bad: Status 404 returned error can't find the container with id 0a0ead4e384099d2650c4921134cca18e340982ee4f30c918f8de42fcf834bad
Nov 02 12:35:23.568035 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.567998    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.568137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.568077    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.568137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.568118    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:23.580423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:23.580375    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podStartSLOduration=1.580361897 podStartE2EDuration="1.580361897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:23.578269234 +0000 UTC m=+521.227323213" watchObservedRunningTime="2026-11-02 12:35:23.580361897 +0000 UTC m=+521.229415854"
Nov 02 12:35:24.074500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.074458    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:24.080160 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.080127    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:24.538194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.538155    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerStarted","Data":"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"}
Nov 02 12:35:24.538389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.538210    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerStarted","Data":"0a0ead4e384099d2650c4921134cca18e340982ee4f30c918f8de42fcf834bad"}
Nov 02 12:35:24.538462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.538434    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:24.539268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.539246    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:24.589059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:24.589007    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podStartSLOduration=2.588993988 podStartE2EDuration="2.588993988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:24.586418535 +0000 UTC m=+522.235472513" watchObservedRunningTime="2026-11-02 12:35:24.588993988 +0000 UTC m=+522.238047942"
Nov 02 12:35:25.232386 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:25.232295    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:25.238628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:25.238600    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:25.545405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:25.545370    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:25.546190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:25.546167    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:27.118489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.118453    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:27.122332 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.122310    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.149027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.148998    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:27.153078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.153054    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d9dc4ff4\""
Nov 02 12:35:27.175321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.175293    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d9dc4ff4-dockercfg-6f8cs\""
Nov 02 12:35:27.194108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.194071    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:27.197860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.197821    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.218576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.218550    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:27.229221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.229185    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.229463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.229438    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.229576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.229482    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhsg\" (UniqueName: \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.229576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.229560    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpqd\" (UniqueName: \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.330233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.330187    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.330410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.330335    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.330410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.330365    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhsg\" (UniqueName: \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.330552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.330431    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpqd\" (UniqueName: \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.331189 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.331166    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.331278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.331185    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.355173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.355138    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpqd\" (UniqueName: \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\") pod \"payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.355173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.355165    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhsg\" (UniqueName: \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\") pod \"payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.437886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.437761    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:27.514136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.514102    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:27.759251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.752040    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:27.823770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:27.823741    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:27.825805 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:27.825779    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b7504_669a_46ec_bc57_9df3230f6f2a.slice/crio-5daa15a4806f939dc89eba77d2421ebb425bb650e6bdbc4553ff74aa1dcc5178 WatchSource:0}: Error finding container 5daa15a4806f939dc89eba77d2421ebb425bb650e6bdbc4553ff74aa1dcc5178: Status 404 returned error can't find the container with id 5daa15a4806f939dc89eba77d2421ebb425bb650e6bdbc4553ff74aa1dcc5178
Nov 02 12:35:28.566985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.566949    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" event={"ID":"3138f133-11a4-4ba3-bf98-09389c5256af","Type":"ContainerStarted","Data":"f143b12bbd65ccafdf3a256a963c4c4699879f1ff78778332f9e1f4f8fd192a8"}
Nov 02 12:35:28.568756 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.568720    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" event={"ID":"940b7504-669a-46ec-bc57-9df3230f6f2a","Type":"ContainerStarted","Data":"5daa15a4806f939dc89eba77d2421ebb425bb650e6bdbc4553ff74aa1dcc5178"}
Nov 02 12:35:28.937186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.933914    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:28.939293 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.938426    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:28.959310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.959281    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:28.968822 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.968777    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-53b6603d\""
Nov 02 12:35:28.969508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:28.969471    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-53b6603d-dockercfg-nrsv8\""
Nov 02 12:35:29.027143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.027093    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:29.034174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.034130    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.059849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.058036    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:29.086871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.066451    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.086871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.066918    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbnz\" (UniqueName: \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.177093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.177050    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.177266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.177178    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbnz\" (UniqueName: \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.177266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.177225    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcgs\" (UniqueName: \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.177385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.177322    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.178414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.178378    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.202939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.202866    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbnz\" (UniqueName: \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\") pod \"payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.259112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.259074    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:29.278952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.278896    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.279103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.279002    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcgs\" (UniqueName: \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.280727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.279777    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.306307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.306192    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcgs\" (UniqueName: \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\") pod \"payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") " pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.402245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.402009    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:29.601863 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:29.598776    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21b1caf_a4a5_4b9a_b7d4_b66469f465b5.slice/crio-f4df2e81051d062a303db7f1357533218ed565b894c4edce150f31c7bbc4f246 WatchSource:0}: Error finding container f4df2e81051d062a303db7f1357533218ed565b894c4edce150f31c7bbc4f246: Status 404 returned error can't find the container with id f4df2e81051d062a303db7f1357533218ed565b894c4edce150f31c7bbc4f246
Nov 02 12:35:29.601863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.601111    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:29.739151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.739027    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:29.747062 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:29.747002    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda74cec5_9e36_4758_9a26_ea88d189c707.slice/crio-e610648e628a5b946c038cc786793df946f866330c3f5627b9844418bfb15e33 WatchSource:0}: Error finding container e610648e628a5b946c038cc786793df946f866330c3f5627b9844418bfb15e33: Status 404 returned error can't find the container with id e610648e628a5b946c038cc786793df946f866330c3f5627b9844418bfb15e33
Nov 02 12:35:29.981492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.981441    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:35:29.986127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:29.986099    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.009883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.009853    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:35:30.018640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.016177    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f080cb5f-dockercfg-jf58p\""
Nov 02 12:35:30.070887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.064122    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f080cb5f\""
Nov 02 12:35:30.093404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.093362    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqnkv\" (UniqueName: \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.093583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.093515    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.195101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.195056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.195261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.195163    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqnkv\" (UniqueName: \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.196499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.196466    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.288402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.288295    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqnkv\" (UniqueName: \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\") pod \"payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") " pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.310237 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.310041    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:30.598240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.598106    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" event={"ID":"da74cec5-9e36-4758-9a26-ea88d189c707","Type":"ContainerStarted","Data":"e610648e628a5b946c038cc786793df946f866330c3f5627b9844418bfb15e33"}
Nov 02 12:35:30.600428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.600353    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" event={"ID":"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5","Type":"ContainerStarted","Data":"f4df2e81051d062a303db7f1357533218ed565b894c4edce150f31c7bbc4f246"}
Nov 02 12:35:30.698125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:30.698090    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:35:31.528700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.528404    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:35:31.533374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.533352    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.557017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.556512    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:35:31.607917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.607877    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" event={"ID":"0c41af0d-a06e-4304-9972-3179f1ec11b8","Type":"ContainerStarted","Data":"9c789e73cd58f50b034c1b9899228a28f4c294811545c19c90333bfe5e56ffcd"}
Nov 02 12:35:31.608066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.607955    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" event={"ID":"0c41af0d-a06e-4304-9972-3179f1ec11b8","Type":"ContainerStarted","Data":"d963a0dedb4580906509ccceecebee1888b0b07cdd141577f17e811d853b78fd"}
Nov 02 12:35:31.608125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.608089    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:31.609611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.609591    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" event={"ID":"3138f133-11a4-4ba3-bf98-09389c5256af","Type":"ContainerStarted","Data":"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"}
Nov 02 12:35:31.609724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.609669    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:31.611421 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.611402    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" event={"ID":"940b7504-669a-46ec-bc57-9df3230f6f2a","Type":"ContainerStarted","Data":"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"}
Nov 02 12:35:31.611543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.611455    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:31.621289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621263    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621316    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621376    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621502    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621554    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621834    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621942    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.621996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.621981    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvww8\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.622089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.622034    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.675035 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.667456    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2f0bef09-openshift-default-dockercfg-qtk5v\""
Nov 02 12:35:31.681978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.675888    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" podStartSLOduration=1.986459549 podStartE2EDuration="2.675864335s" totalImagesPullingTime="689.404786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:31.667735582 +0000 UTC m=+529.316789541" watchObservedRunningTime="2026-11-02 12:35:31.675864335 +0000 UTC m=+529.324918294"
Nov 02 12:35:31.720933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.720881    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" podStartSLOduration=1.261480792 podStartE2EDuration="4.720865987s" totalImagesPullingTime="3.459385195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:31.716400033 +0000 UTC m=+529.365454003" watchObservedRunningTime="2026-11-02 12:35:31.720865987 +0000 UTC m=+529.369919945"
Nov 02 12:35:31.722816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.722786    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.722940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.722874    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.722963    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723006    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723100    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723235    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723316    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvww8\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723356    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723455 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723408    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723631    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723734    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723734    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.723936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.723911    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.724075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.724059    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.725867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.725787    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.726025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.726007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.756487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.756461    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvww8\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.759969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.759943    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\") pod \"e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") " pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:31.776807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.776762    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" podStartSLOduration=1.334199478 podStartE2EDuration="4.776749901s" totalImagesPullingTime="3.442550423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:31.775508503 +0000 UTC m=+529.424562462" watchObservedRunningTime="2026-11-02 12:35:31.776749901 +0000 UTC m=+529.425803859"
Nov 02 12:35:31.851544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:31.851506    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:32.173176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.173143    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:35:32.175902 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:32.175873    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a94e41_147d_4ac0_907d_45fc6480e232.slice/crio-06fb5e9a783c9707706521ede005a704f2c75d4dfc2996ca92d260a87b181536 WatchSource:0}: Error finding container 06fb5e9a783c9707706521ede005a704f2c75d4dfc2996ca92d260a87b181536: Status 404 returned error can't find the container with id 06fb5e9a783c9707706521ede005a704f2c75d4dfc2996ca92d260a87b181536
Nov 02 12:35:32.178553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.178521    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:32.178646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.178586    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:32.178646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.178617    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:32.618870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.618816    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerStarted","Data":"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"}
Nov 02 12:35:32.619085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.618900    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerStarted","Data":"06fb5e9a783c9707706521ede005a704f2c75d4dfc2996ca92d260a87b181536"}
Nov 02 12:35:32.620578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.620546    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" event={"ID":"da74cec5-9e36-4758-9a26-ea88d189c707","Type":"ContainerStarted","Data":"77f51aee1078db46eda8fb0d30fdfa351f4de61efd8da46eabbfe73a3073ed34"}
Nov 02 12:35:32.620717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.620659    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:32.622270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.622242    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" event={"ID":"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5","Type":"ContainerStarted","Data":"386ac76f59e45187a682b0629493f6a365a0748b027a355c110a66897787d9c2"}
Nov 02 12:35:32.622434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.622413    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:32.668292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.668222    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podStartSLOduration=1.6682033330000001 podStartE2EDuration="1.668203333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:32.667669014 +0000 UTC m=+530.316722973" watchObservedRunningTime="2026-11-02 12:35:32.668203333 +0000 UTC m=+530.317257292"
Nov 02 12:35:32.712262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.712205    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" podStartSLOduration=1.7732455150000002 podStartE2EDuration="3.712192662s" totalImagesPullingTime="1.938947147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:32.71002919 +0000 UTC m=+530.359083149" watchObservedRunningTime="2026-11-02 12:35:32.712192662 +0000 UTC m=+530.361246619"
Nov 02 12:35:32.755466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.755398    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" podStartSLOduration=2.823154155 podStartE2EDuration="4.755379063s" totalImagesPullingTime="1.932224908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:32.75223692 +0000 UTC m=+530.401290881" watchObservedRunningTime="2026-11-02 12:35:32.755379063 +0000 UTC m=+530.404433021"
Nov 02 12:35:32.851957 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:32.851916    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:33.858855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:33.858808    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:34.635869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:34.633810    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:34.635869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:34.634865    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:39.760308 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:35:39.760271    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:35:40.789949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.789899    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"]
Nov 02 12:35:40.795211 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.795190    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:40.813543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.813373    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"]
Nov 02 12:35:40.823908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.823880    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:35:40.824038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.823987    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:35:40.824104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.824086    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:35:40.824181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.824169    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:35:40.896614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.896569    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlj8v\" (UniqueName: \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:40.896776 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.896690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:40.896914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:40.896890    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.005558 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.005506    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.006481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.006456    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.006661 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.006641    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlj8v\" (UniqueName: \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.007691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.007660    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.009788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.009767    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.049046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.048955    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlj8v\" (UniqueName: \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\") pod \"maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.109805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.109766    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:41.459195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.459162    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"]
Nov 02 12:35:41.462799 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:41.462749    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8343545a_d544_44bd_a530_5c1c5593c914.slice/crio-93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5 WatchSource:0}: Error finding container 93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5: Status 404 returned error can't find the container with id 93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5
Nov 02 12:35:41.706238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:41.706201    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq" event={"ID":"8343545a-d544-44bd-a530-5c1c5593c914","Type":"ContainerStarted","Data":"93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5"}
Nov 02 12:35:42.270870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.263476    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"]
Nov 02 12:35:42.270870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.268392    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.290699 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.290660    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"]
Nov 02 12:35:42.327313 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.327264    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.327464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.327365    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5qx\" (UniqueName: \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.327464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.327406    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.436854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.428613    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.436854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.428682    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5qx\" (UniqueName: \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.436854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.428711    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.436854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.430048    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.440861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.439316    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.454959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.454927    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5qx\" (UniqueName: \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\") pod \"maas-api-revoke-keys-e2e-ait-53b6603d-j72cf\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.587697 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.587624    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:42.625844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.625801    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:42.628403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.628378    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:42.629503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.629023    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:35:42.935076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:42.934993    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"]
Nov 02 12:35:42.937481 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:42.937447    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369d19bf_141f_455e_ab15_10e9c41bc877.slice/crio-0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2 WatchSource:0}: Error finding container 0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2: Status 404 returned error can't find the container with id 0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2
Nov 02 12:35:43.631046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:43.631010    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:43.632280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:43.632259    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:43.735183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:43.735098    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf" event={"ID":"369d19bf-141f-455e-ab15-10e9c41bc877","Type":"ContainerStarted","Data":"0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2"}
Nov 02 12:35:45.754010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.753955    4928 generic.go:356] "Generic (PLEG): container finished" podID="8343545a-d544-44bd-a530-5c1c5593c914" containerID="3ac6fb0525cd7d88e74344c07cfa5b4b67d8581bd7aa621dcc7926fa8c6efc63" exitCode=0
Nov 02 12:35:45.754533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.754081    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq" event={"ID":"8343545a-d544-44bd-a530-5c1c5593c914","Type":"ContainerDied","Data":"3ac6fb0525cd7d88e74344c07cfa5b4b67d8581bd7aa621dcc7926fa8c6efc63"}
Nov 02 12:35:45.756872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.756820    4928 generic.go:356] "Generic (PLEG): container finished" podID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerID="b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf" exitCode=0
Nov 02 12:35:45.756872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.756857    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerDied","Data":"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"}
Nov 02 12:35:45.757294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.757278    4928 scope.go:122] "RemoveContainer" containerID="b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"
Nov 02 12:35:45.758884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.758861    4928 generic.go:356] "Generic (PLEG): container finished" podID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerID="2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a" exitCode=0
Nov 02 12:35:45.758968 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.758932    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerDied","Data":"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"}
Nov 02 12:35:45.759053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.758926    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.759133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.759112    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.759190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.759164    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.759353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.759334    4928 scope.go:122] "RemoveContainer" containerID="2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"
Nov 02 12:35:45.760776 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.760745    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.760891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.760859    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.760943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.760907    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:45.762995 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.762938    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf" event={"ID":"369d19bf-141f-455e-ab15-10e9c41bc877","Type":"ContainerStarted","Data":"d790dfff60630baf472aea3bca9eb009d19c86e8dab2477217fa086a14877821"}
Nov 02 12:35:45.851701 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:45.851638    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf" podStartSLOduration=2.101831662 podStartE2EDuration="3.851616058s" totalImagesPullingTime="1.749784396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:45.84940422 +0000 UTC m=+543.498458178" watchObservedRunningTime="2026-11-02 12:35:45.851616058 +0000 UTC m=+543.500670017"
Nov 02 12:35:46.790321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:46.790285    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerStarted","Data":"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"}
Nov 02 12:35:46.794089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:46.794063    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerStarted","Data":"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"}
Nov 02 12:35:46.796121 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:46.796091    4928 generic.go:356] "Generic (PLEG): container finished" podID="369d19bf-141f-455e-ab15-10e9c41bc877" containerID="d790dfff60630baf472aea3bca9eb009d19c86e8dab2477217fa086a14877821" exitCode=0
Nov 02 12:35:46.796260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:46.796239    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf" event={"ID":"369d19bf-141f-455e-ab15-10e9c41bc877","Type":"ContainerDied","Data":"d790dfff60630baf472aea3bca9eb009d19c86e8dab2477217fa086a14877821"}
Nov 02 12:35:47.084641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.084543    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:47.085245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.084969    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:47.085245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.085031    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:47.090626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.090606    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:47.208221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.208187    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\") pod \"8343545a-d544-44bd-a530-5c1c5593c914\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") "
Nov 02 12:35:47.208423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.208302    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\") pod \"8343545a-d544-44bd-a530-5c1c5593c914\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") "
Nov 02 12:35:47.208423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.208382    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\" (UniqueName: \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\") pod \"8343545a-d544-44bd-a530-5c1c5593c914\" (UID: \"8343545a-d544-44bd-a530-5c1c5593c914\") "
Nov 02 12:35:47.208958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.208912    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca" pod "8343545a-d544-44bd-a530-5c1c5593c914" (UID: "8343545a-d544-44bd-a530-5c1c5593c914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:47.210842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.210775    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca" pod "8343545a-d544-44bd-a530-5c1c5593c914" (UID: "8343545a-d544-44bd-a530-5c1c5593c914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:47.211195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.211169    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v" pod "8343545a-d544-44bd-a530-5c1c5593c914" (UID: "8343545a-d544-44bd-a530-5c1c5593c914"). InnerVolumeSpecName "kube-api-access-zlj8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:47.238178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.238147    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:47.262535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.262497    4928 patch_prober.go:29] interesting pod/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 12:35:47.262793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.262760    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 12:35:47.311364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.309511    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:47.311364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.309537    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8343545a-d544-44bd-a530-5c1c5593c914-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:47.311364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.309546    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlj8v\" (UniqueName: \"kubernetes.io/projected/8343545a-d544-44bd-a530-5c1c5593c914-kube-api-access-zlj8v\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:47.803960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.803921    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"
Nov 02 12:35:47.804434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.804403    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq" event={"ID":"8343545a-d544-44bd-a530-5c1c5593c914","Type":"ContainerDied","Data":"93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5"}
Nov 02 12:35:47.804507 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:47.804463    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c49ae263b19daa5f6af9391b9b0c5ea0dda2944cd784a5600c96b5fb6b36f5"
Nov 02 12:35:48.114938 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.106280    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:48.114938 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.106349    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:48.312630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.312598    4928 patch_prober.go:29] interesting pod/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 12:35:48.312891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.312861    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 12:35:48.397234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.397133    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:48.485232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.485188    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\") pod \"369d19bf-141f-455e-ab15-10e9c41bc877\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") "
Nov 02 12:35:48.508857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.485532    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\" (UniqueName: \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\") pod \"369d19bf-141f-455e-ab15-10e9c41bc877\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") "
Nov 02 12:35:48.510103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.509618    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca" pod "369d19bf-141f-455e-ab15-10e9c41bc877" (UID: "369d19bf-141f-455e-ab15-10e9c41bc877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:48.510766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.510374    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\") pod \"369d19bf-141f-455e-ab15-10e9c41bc877\" (UID: \"369d19bf-141f-455e-ab15-10e9c41bc877\") "
Nov 02 12:35:48.537227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.537194    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:48.600849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.595609    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx" pod "369d19bf-141f-455e-ab15-10e9c41bc877" (UID: "369d19bf-141f-455e-ab15-10e9c41bc877"). InnerVolumeSpecName "kube-api-access-vg5qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:48.636012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.627487    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca" pod "369d19bf-141f-455e-ab15-10e9c41bc877" (UID: "369d19bf-141f-455e-ab15-10e9c41bc877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:48.641563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.641485    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5qx\" (UniqueName: \"kubernetes.io/projected/369d19bf-141f-455e-ab15-10e9c41bc877-kube-api-access-vg5qx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:48.641563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.641528    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369d19bf-141f-455e-ab15-10e9c41bc877-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:48.814570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.814532    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf" event={"ID":"369d19bf-141f-455e-ab15-10e9c41bc877","Type":"ContainerDied","Data":"0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2"}
Nov 02 12:35:48.815164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.815142    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a49bd4c4a5ea12dd06e9d95138691e6c416b526ffa54623a238bdc33b2ac3a2"
Nov 02 12:35:48.815355 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:48.815341    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"
Nov 02 12:35:49.074361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:49.074259    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:49.074645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:49.074607    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:49.233916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:49.233143    4928 patch_prober.go:29] interesting pod/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 12:35:49.233916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:49.233210    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 12:35:50.077294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:50.077254    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:50.077944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:50.077910    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:50.234851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:50.233818    4928 patch_prober.go:29] interesting pod/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 12:35:50.234851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:50.233907    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 12:35:51.077381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.077333    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:51.078063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.078008    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:51.238540 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.238500    4928 patch_prober.go:29] interesting pod/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 12:35:51.238795 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.238770    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 12:35:51.277857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.261469    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:51.298443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:51.298409    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:52.081849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.081233    4928 patch_prober.go:29] interesting pod/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 12:35:52.081849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.081297    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 12:35:52.380683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.380410    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:52.381059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.381036    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:52.382223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.382181    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:35:52.761133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.761101    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:52.811704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:52.811674    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:53.079000 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.078918    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:53.079238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.079216    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:53.079995 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.079974    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:35:53.899416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899335    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"]
Nov 02 12:35:53.899878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899794    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8343545a-d544-44bd-a530-5c1c5593c914" containerName="revoke-keys"
Nov 02 12:35:53.899878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899806    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8343545a-d544-44bd-a530-5c1c5593c914" containerName="revoke-keys"
Nov 02 12:35:53.899878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899813    4928 cpu_manager.go:414] "containerMap: removing container" podUID="369d19bf-141f-455e-ab15-10e9c41bc877" containerName="revoke-keys"
Nov 02 12:35:53.899878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899819    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369d19bf-141f-455e-ab15-10e9c41bc877" containerName="revoke-keys"
Nov 02 12:35:53.900011 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899924    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8343545a-d544-44bd-a530-5c1c5593c914" containerName="revoke-keys"
Nov 02 12:35:53.900011 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.899933    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="369d19bf-141f-455e-ab15-10e9c41bc877" containerName="revoke-keys"
Nov 02 12:35:53.904997 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.904974    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:53.923665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.923637    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"]
Nov 02 12:35:53.933224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.933201    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:35:53.933371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.933270    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:35:53.933371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.933266    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:35:53.933480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:53.933465    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:35:54.016618 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.016576    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.016799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.016622    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrs9\" (UniqueName: \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.017182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.017155    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.118509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.118472    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrs9\" (UniqueName: \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.118683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.118585    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.118683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.118667    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.119497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.119458    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.121057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.121033    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.143724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.143693    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrs9\" (UniqueName: \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\") pod \"maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.219239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.219161    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:54.538613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.538581    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"]
Nov 02 12:35:54.540362 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:54.540334    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789821d8_c3e9_4093_8ae0_31b286fb452d.slice/crio-3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4 WatchSource:0}: Error finding container 3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4: Status 404 returned error can't find the container with id 3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4
Nov 02 12:35:54.925439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.925396    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl" event={"ID":"789821d8-c3e9-4093-8ae0-31b286fb452d","Type":"ContainerStarted","Data":"6a51337a2056212ebff54ff7d0c266a46430b97bcebca4bb5f050d998f7b1a66"}
Nov 02 12:35:54.926009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.925495    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl" event={"ID":"789821d8-c3e9-4093-8ae0-31b286fb452d","Type":"ContainerStarted","Data":"3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4"}
Nov 02 12:35:54.970413 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:54.970346    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl" podStartSLOduration=1.970331074 podStartE2EDuration="1.970331074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:54.968514391 +0000 UTC m=+552.617568354" watchObservedRunningTime="2026-11-02 12:35:54.970331074 +0000 UTC m=+552.619385032"
Nov 02 12:35:55.788658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:55.788618    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"]
Nov 02 12:35:55.802899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:55.802867    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9dc4ff4-gjpgq"]
Nov 02 12:35:55.942760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:55.942720    4928 generic.go:356] "Generic (PLEG): container finished" podID="789821d8-c3e9-4093-8ae0-31b286fb452d" containerID="6a51337a2056212ebff54ff7d0c266a46430b97bcebca4bb5f050d998f7b1a66" exitCode=0
Nov 02 12:35:55.943278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:55.942939    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl" event={"ID":"789821d8-c3e9-4093-8ae0-31b286fb452d","Type":"ContainerDied","Data":"6a51337a2056212ebff54ff7d0c266a46430b97bcebca4bb5f050d998f7b1a66"}
Nov 02 12:35:56.278928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.278887    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" podUID="940b7504-669a-46ec-bc57-9df3230f6f2a" containerName="payload-processing" containerID="cri-o://9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17" gracePeriod=25
Nov 02 12:35:56.305885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.305846    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" podUID="3138f133-11a4-4ba3-bf98-09389c5256af" containerName="payload-pre-processing" containerID="cri-o://0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8" gracePeriod=25
Nov 02 12:35:56.503217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.503175    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8343545a-d544-44bd-a530-5c1c5593c914" path="/var/lib/kubelet/pods/8343545a-d544-44bd-a530-5c1c5593c914/volumes"
Nov 02 12:35:56.570713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.570685    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:56.606508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.606470    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:56.657052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.657022    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\" (UniqueName: \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\") pod \"940b7504-669a-46ec-bc57-9df3230f6f2a\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") "
Nov 02 12:35:56.657239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.657154    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\") pod \"940b7504-669a-46ec-bc57-9df3230f6f2a\" (UID: \"940b7504-669a-46ec-bc57-9df3230f6f2a\") "
Nov 02 12:35:56.657582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.657555    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume" pod "940b7504-669a-46ec-bc57-9df3230f6f2a" (UID: "940b7504-669a-46ec-bc57-9df3230f6f2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:56.660966 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.660936    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd" pod "940b7504-669a-46ec-bc57-9df3230f6f2a" (UID: "940b7504-669a-46ec-bc57-9df3230f6f2a"). InnerVolumeSpecName "kube-api-access-vgpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:56.758642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.758608    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\") pod \"3138f133-11a4-4ba3-bf98-09389c5256af\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") "
Nov 02 12:35:56.758835 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.758694    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\" (UniqueName: \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\") pod \"3138f133-11a4-4ba3-bf98-09389c5256af\" (UID: \"3138f133-11a4-4ba3-bf98-09389c5256af\") "
Nov 02 12:35:56.759041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.759017    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume" pod "3138f133-11a4-4ba3-bf98-09389c5256af" (UID: "3138f133-11a4-4ba3-bf98-09389c5256af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:56.759125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.759108    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b7504-669a-46ec-bc57-9df3230f6f2a-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:56.759125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.759121    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3138f133-11a4-4ba3-bf98-09389c5256af-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:56.759125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.759130    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgpqd\" (UniqueName: \"kubernetes.io/projected/940b7504-669a-46ec-bc57-9df3230f6f2a-kube-api-access-vgpqd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:56.760785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.760759    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg" pod "3138f133-11a4-4ba3-bf98-09389c5256af" (UID: "3138f133-11a4-4ba3-bf98-09389c5256af"). InnerVolumeSpecName "kube-api-access-zjhsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:56.801570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.801485    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:35:56.860068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.860029    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhsg\" (UniqueName: \"kubernetes.io/projected/3138f133-11a4-4ba3-bf98-09389c5256af-kube-api-access-zjhsg\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:56.950734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.950699    4928 generic.go:356] "Generic (PLEG): container finished" podID="3138f133-11a4-4ba3-bf98-09389c5256af" containerID="0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8" exitCode=0
Nov 02 12:35:56.951259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.950737    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"
Nov 02 12:35:56.951259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.950777    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" event={"ID":"3138f133-11a4-4ba3-bf98-09389c5256af","Type":"ContainerDied","Data":"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"}
Nov 02 12:35:56.951259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.950876    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl" event={"ID":"3138f133-11a4-4ba3-bf98-09389c5256af","Type":"ContainerDied","Data":"f143b12bbd65ccafdf3a256a963c4c4699879f1ff78778332f9e1f4f8fd192a8"}
Nov 02 12:35:56.951259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.950915    4928 scope.go:122] "RemoveContainer" containerID="0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"
Nov 02 12:35:56.952981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.952959    4928 generic.go:356] "Generic (PLEG): container finished" podID="940b7504-669a-46ec-bc57-9df3230f6f2a" containerID="9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17" exitCode=0
Nov 02 12:35:56.953265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.953009    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"
Nov 02 12:35:56.961495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.953026    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" event={"ID":"940b7504-669a-46ec-bc57-9df3230f6f2a","Type":"ContainerDied","Data":"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"}
Nov 02 12:35:56.961495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.953622    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x" event={"ID":"940b7504-669a-46ec-bc57-9df3230f6f2a","Type":"ContainerDied","Data":"5daa15a4806f939dc89eba77d2421ebb425bb650e6bdbc4553ff74aa1dcc5178"}
Nov 02 12:35:56.968577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.968549    4928 scope.go:122] "RemoveContainer" containerID="0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"
Nov 02 12:35:56.969001 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:35:56.968970    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8\": container with ID starting with 0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8 not found: ID does not exist" containerID="0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"
Nov 02 12:35:56.969118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.969007    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8"} err="failed to get container status \"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8\": rpc error: code = NotFound desc = could not find container \"0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8\": container with ID starting with 0de7a6ba68214b96de0583e29d29a31cb2d9b511cb3aee51a101956495e67ae8 not found: ID does not exist"
Nov 02 12:35:56.969118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.969032    4928 scope.go:122] "RemoveContainer" containerID="9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"
Nov 02 12:35:56.988023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.987705    4928 scope.go:122] "RemoveContainer" containerID="9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"
Nov 02 12:35:56.988341 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:35:56.988258    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17\": container with ID starting with 9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17 not found: ID does not exist" containerID="9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"
Nov 02 12:35:56.988341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:56.988299    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17"} err="failed to get container status \"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17\": rpc error: code = NotFound desc = could not find container \"9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17\": container with ID starting with 9be59d4dca5761da9f156bfbc862aaf38a805fc817a85f3c948cc15088be6c17 not found: ID does not exist"
Nov 02 12:35:57.028855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.028793    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:57.033144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.032957    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9dc4ff4-7cd89664f9-t52bl"]
Nov 02 12:35:57.072293 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.072160    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:57.079902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.079862    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9dc4ff4-84f86c455c-jxx6x"]
Nov 02 12:35:57.126734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.126703    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:57.227460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.227428    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:35:57.228115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228099    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3138f133-11a4-4ba3-bf98-09389c5256af" containerName="payload-pre-processing"
Nov 02 12:35:57.228175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228118    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3138f133-11a4-4ba3-bf98-09389c5256af" containerName="payload-pre-processing"
Nov 02 12:35:57.228175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228140    4928 cpu_manager.go:414] "containerMap: removing container" podUID="789821d8-c3e9-4093-8ae0-31b286fb452d" containerName="revoke-keys"
Nov 02 12:35:57.228175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228149    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789821d8-c3e9-4093-8ae0-31b286fb452d" containerName="revoke-keys"
Nov 02 12:35:57.228175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228166    4928 cpu_manager.go:414] "containerMap: removing container" podUID="940b7504-669a-46ec-bc57-9df3230f6f2a" containerName="payload-processing"
Nov 02 12:35:57.228175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228174    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b7504-669a-46ec-bc57-9df3230f6f2a" containerName="payload-processing"
Nov 02 12:35:57.228334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228268    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3138f133-11a4-4ba3-bf98-09389c5256af" containerName="payload-pre-processing"
Nov 02 12:35:57.228334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228281    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="789821d8-c3e9-4093-8ae0-31b286fb452d" containerName="revoke-keys"
Nov 02 12:35:57.228334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.228292    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b7504-669a-46ec-bc57-9df3230f6f2a" containerName="payload-processing"
Nov 02 12:35:57.231950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.231924    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.254101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.254074    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-068756-openshift-default-dockercfg-kp7kc\""
Nov 02 12:35:57.255814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.255791    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:35:57.263459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.263405    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\") pod \"789821d8-c3e9-4093-8ae0-31b286fb452d\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") "
Nov 02 12:35:57.263568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.263481    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\") pod \"789821d8-c3e9-4093-8ae0-31b286fb452d\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") "
Nov 02 12:35:57.263568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.263540    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\" (UniqueName: \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\") pod \"789821d8-c3e9-4093-8ae0-31b286fb452d\" (UID: \"789821d8-c3e9-4093-8ae0-31b286fb452d\") "
Nov 02 12:35:57.263864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.263802    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca" pod "789821d8-c3e9-4093-8ae0-31b286fb452d" (UID: "789821d8-c3e9-4093-8ae0-31b286fb452d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:57.265652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.265630    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca" pod "789821d8-c3e9-4093-8ae0-31b286fb452d" (UID: "789821d8-c3e9-4093-8ae0-31b286fb452d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:57.265749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.265711    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9" pod "789821d8-c3e9-4093-8ae0-31b286fb452d" (UID: "789821d8-c3e9-4093-8ae0-31b286fb452d"). InnerVolumeSpecName "kube-api-access-vzrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:57.365049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.364942    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365022    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365105    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365152    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365212    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365285    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365337    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365365    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365393    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjht\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365493    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365540    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789821d8-c3e9-4093-8ae0-31b286fb452d-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.365586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.365556    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrs9\" (UniqueName: \"kubernetes.io/projected/789821d8-c3e9-4093-8ae0-31b286fb452d-kube-api-access-vzrs9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.466768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.466726    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.466976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.466795    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.466976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.466850    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.466976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.466896    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.466976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.466928    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467017    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467046    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467080    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467112    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjht\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467421 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467318    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467421 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467349    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467479    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467549 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467512    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.467727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.467703    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.469216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.469193    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.469484 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.469466    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.491683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.491646    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.491839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.491807    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjht\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.549312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.549271    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:57.761967 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.761913    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" podUID="da74cec5-9e36-4758-9a26-ea88d189c707" containerName="payload-processing" containerID="cri-o://77f51aee1078db46eda8fb0d30fdfa351f4de61efd8da46eabbfe73a3073ed34" gracePeriod=25
Nov 02 12:35:57.812730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.812665    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" podUID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" containerName="payload-pre-processing" containerID="cri-o://386ac76f59e45187a682b0629493f6a365a0748b027a355c110a66897787d9c2" gracePeriod=25
Nov 02 12:35:57.916700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.916669    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:35:57.918870 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:35:57.918840    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3443c7_8492_4c37_8927_3e2315d3af30.slice/crio-4bf22485e6f41195bce664945edb8472ee25f7392b8c1794faacd882c7c69ff8 WatchSource:0}: Error finding container 4bf22485e6f41195bce664945edb8472ee25f7392b8c1794faacd882c7c69ff8: Status 404 returned error can't find the container with id 4bf22485e6f41195bce664945edb8472ee25f7392b8c1794faacd882c7c69ff8
Nov 02 12:35:57.921356 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.921318    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:57.921463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.921405    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:57.921463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.921452    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:57.965534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.965354    4928 generic.go:356] "Generic (PLEG): container finished" podID="da74cec5-9e36-4758-9a26-ea88d189c707" containerID="77f51aee1078db46eda8fb0d30fdfa351f4de61efd8da46eabbfe73a3073ed34" exitCode=0
Nov 02 12:35:57.965534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.965468    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" event={"ID":"da74cec5-9e36-4758-9a26-ea88d189c707","Type":"ContainerDied","Data":"77f51aee1078db46eda8fb0d30fdfa351f4de61efd8da46eabbfe73a3073ed34"}
Nov 02 12:35:57.969913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.969889    4928 generic.go:356] "Generic (PLEG): container finished" podID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" containerID="386ac76f59e45187a682b0629493f6a365a0748b027a355c110a66897787d9c2" exitCode=0
Nov 02 12:35:57.970234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.970042    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" event={"ID":"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5","Type":"ContainerDied","Data":"386ac76f59e45187a682b0629493f6a365a0748b027a355c110a66897787d9c2"}
Nov 02 12:35:57.972431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.971311    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl" event={"ID":"789821d8-c3e9-4093-8ae0-31b286fb452d","Type":"ContainerDied","Data":"3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4"}
Nov 02 12:35:57.972431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.971371    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3108146f01670aa4e49379843eea998c43ddc1164f32c1b3c9c006423d6a43b4"
Nov 02 12:35:57.972431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.971377    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"
Nov 02 12:35:57.973371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.973341    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerStarted","Data":"4bf22485e6f41195bce664945edb8472ee25f7392b8c1794faacd882c7c69ff8"}
Nov 02 12:35:57.975337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.975312    4928 generic.go:356] "Generic (PLEG): container finished" podID="40a94e41-147d-4ac0-907d-45fc6480e232" containerID="21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5" exitCode=0
Nov 02 12:35:57.975447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.975380    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerDied","Data":"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"}
Nov 02 12:35:57.975781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.975764    4928 scope.go:122] "RemoveContainer" containerID="21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"
Nov 02 12:35:57.978508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.977899    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:57.978508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.977984    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:57.978508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:57.978030    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:35:58.010191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.010165    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"]
Nov 02 12:35:58.013414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.013349    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-53b6603d-j72cf"]
Nov 02 12:35:58.093165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.092766    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:58.174672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.173761    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\") pod \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") "
Nov 02 12:35:58.174672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.173900    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\" (UniqueName: \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\") pod \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\" (UID: \"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5\") "
Nov 02 12:35:58.176399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.176219    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume" pod "b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" (UID: "b21b1caf-a4a5-4b9a-b7d4-b66469f465b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:58.184595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.184272    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:58.265806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.265751    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz" pod "b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" (UID: "b21b1caf-a4a5-4b9a-b7d4-b66469f465b5"). InnerVolumeSpecName "kube-api-access-vbbnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:58.276633 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.276596    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\") pod \"da74cec5-9e36-4758-9a26-ea88d189c707\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") "
Nov 02 12:35:58.276814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.276703    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\" (UniqueName: \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\") pod \"da74cec5-9e36-4758-9a26-ea88d189c707\" (UID: \"da74cec5-9e36-4758-9a26-ea88d189c707\") "
Nov 02 12:35:58.277169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.277046    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbnz\" (UniqueName: \"kubernetes.io/projected/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-kube-api-access-vbbnz\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.277169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.277070    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.277169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.277059    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume" pod "da74cec5-9e36-4758-9a26-ea88d189c707" (UID: "da74cec5-9e36-4758-9a26-ea88d189c707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:58.279527 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.279454    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs" pod "da74cec5-9e36-4758-9a26-ea88d189c707" (UID: "da74cec5-9e36-4758-9a26-ea88d189c707"). InnerVolumeSpecName "kube-api-access-ntcgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:58.378445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.378408    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74cec5-9e36-4758-9a26-ea88d189c707-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.378445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.378439    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntcgs\" (UniqueName: \"kubernetes.io/projected/da74cec5-9e36-4758-9a26-ea88d189c707-kube-api-access-ntcgs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.497326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.497285    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3138f133-11a4-4ba3-bf98-09389c5256af" path="/var/lib/kubelet/pods/3138f133-11a4-4ba3-bf98-09389c5256af/volumes"
Nov 02 12:35:58.497734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.497719    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369d19bf-141f-455e-ab15-10e9c41bc877" path="/var/lib/kubelet/pods/369d19bf-141f-455e-ab15-10e9c41bc877/volumes"
Nov 02 12:35:58.498200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.498183    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940b7504-669a-46ec-bc57-9df3230f6f2a" path="/var/lib/kubelet/pods/940b7504-669a-46ec-bc57-9df3230f6f2a/volumes"
Nov 02 12:35:58.567197 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.567157    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:35:58.567517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.567482    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy" containerID="cri-o://c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee" gracePeriod=30
Nov 02 12:35:58.983266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.983183    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"
Nov 02 12:35:58.983266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.983210    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg" event={"ID":"da74cec5-9e36-4758-9a26-ea88d189c707","Type":"ContainerDied","Data":"e610648e628a5b946c038cc786793df946f866330c3f5627b9844418bfb15e33"}
Nov 02 12:35:58.983754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.983279    4928 scope.go:122] "RemoveContainer" containerID="77f51aee1078db46eda8fb0d30fdfa351f4de61efd8da46eabbfe73a3073ed34"
Nov 02 12:35:58.984897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.984867    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb" event={"ID":"b21b1caf-a4a5-4b9a-b7d4-b66469f465b5","Type":"ContainerDied","Data":"f4df2e81051d062a303db7f1357533218ed565b894c4edce150f31c7bbc4f246"}
Nov 02 12:35:58.985016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.984875    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"
Nov 02 12:35:58.987889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.987863    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerStarted","Data":"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"}
Nov 02 12:35:58.990225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.990203    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerStarted","Data":"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"}
Nov 02 12:35:58.995338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:58.995319    4928 scope.go:122] "RemoveContainer" containerID="386ac76f59e45187a682b0629493f6a365a0748b027a355c110a66897787d9c2"
Nov 02 12:35:59.052856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.045116    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:59.052856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.048844    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53b6603d-6dd9cb5cf7-qg2mg"]
Nov 02 12:35:59.157083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.157041    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:59.161573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.161545    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53b6603d-7b76b95c4c-tx4tb"]
Nov 02 12:35:59.204198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.204134    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podStartSLOduration=2.204121334 podStartE2EDuration="2.204121334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:59.202706912 +0000 UTC m=+556.851760870" watchObservedRunningTime="2026-11-02 12:35:59.204121334 +0000 UTC m=+556.853175292"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.481761    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"]
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.489148    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" containerName="payload-pre-processing"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.489177    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" containerName="payload-pre-processing"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.491366    4928 cpu_manager.go:414] "containerMap: removing container" podUID="da74cec5-9e36-4758-9a26-ea88d189c707" containerName="payload-processing"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.491394    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da74cec5-9e36-4758-9a26-ea88d189c707" containerName="payload-processing"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.491652    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="da74cec5-9e36-4758-9a26-ea88d189c707" containerName="payload-processing"
Nov 02 12:35:59.496218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.491677    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" containerName="payload-pre-processing"
Nov 02 12:35:59.502500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.502437    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.503753 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.503723    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"]
Nov 02 12:35:59.551655 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.551613    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:35:59.551908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.551780    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:35:59.551982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.551907    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:35:59.552302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.552279    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:59.552452 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.552437    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:35:59.558159 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.558135    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:35:59.603708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.603671    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:35:59.603986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.603940    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy" containerID="cri-o://4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a" gracePeriod=30
Nov 02 12:35:59.609409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.609381    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.609534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.609419    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prc2f\" (UniqueName: \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.609588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.609566    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.713585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.711449    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.713585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.711505    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prc2f\" (UniqueName: \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.713585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.711618    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.740485 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.739085    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.765774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.765691    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.775688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.775626    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prc2f\" (UniqueName: \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\") pod \"maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.832039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.832000    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:35:59.852379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.852331    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:35:59.852681 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.852653    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:35:59.852751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.852730    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 12:35:59.999771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:35:59.999724    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:00.000803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.000779    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:00.186898 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.180392    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"]
Nov 02 12:36:00.498996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.498962    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21b1caf-a4a5-4b9a-b7d4-b66469f465b5" path="/var/lib/kubelet/pods/b21b1caf-a4a5-4b9a-b7d4-b66469f465b5/volumes"
Nov 02 12:36:00.499414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.499398    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da74cec5-9e36-4758-9a26-ea88d189c707" path="/var/lib/kubelet/pods/da74cec5-9e36-4758-9a26-ea88d189c707/volumes"
Nov 02 12:36:00.852611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.852521    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:36:00.852611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.852589    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 12:36:00.975532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.975493    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:36:00.980148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.980119    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:00.998288 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:00.998264    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843978-openshift-default-dockercfg-nkwwj\""
Nov 02 12:36:01.006564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.006536    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:36:01.009987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.008558    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h" event={"ID":"20f333a6-6ea2-4616-a67a-e17ccb28b375","Type":"ContainerStarted","Data":"27a2941f2cb5a636d45f0bdb36a39c43457fc0e6a804d33c94b0397b9ea4fbf5"}
Nov 02 12:36:01.009987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.008604    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h" event={"ID":"20f333a6-6ea2-4616-a67a-e17ccb28b375","Type":"ContainerStarted","Data":"ddb0acf2d7b3320b9384b1304702fffc6572d72e9c16adfa47fc95a2590803c6"}
Nov 02 12:36:01.056531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056485    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.056717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056629    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.056717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056692    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.056865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056720    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.056865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056756    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdqd\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.056865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056814    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.058059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.056898    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.058059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.057956    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.058270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.058060    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.159708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159601    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.159708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159667    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.159708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159698    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159728    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdqd\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159782    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159820    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159925    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.159994    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.160068    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.160229    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.160870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.160845    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.161262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.161231    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.161596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.161573    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.161734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.161712    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.163394 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.163374    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.163504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.163404    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.185236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.185212    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.185628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.185613    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdqd\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.299338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.299306    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:01.659879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.659801    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h" podStartSLOduration=2.659783145 podStartE2EDuration="2.659783145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:01.104647262 +0000 UTC m=+558.753701222" watchObservedRunningTime="2026-11-02 12:36:01.659783145 +0000 UTC m=+559.308837102"
Nov 02 12:36:01.661359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.661336    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:36:01.663636 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:01.663602    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58dba09_86a0_4bdd_b0a1_ef1431850452.slice/crio-1dbd153c3be291b195073dc903e1a6a42f0ae10526f8785f65a61029d7348692 WatchSource:0}: Error finding container 1dbd153c3be291b195073dc903e1a6a42f0ae10526f8785f65a61029d7348692: Status 404 returned error can't find the container with id 1dbd153c3be291b195073dc903e1a6a42f0ae10526f8785f65a61029d7348692
Nov 02 12:36:01.666044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.666013    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:01.666108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.666076    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:01.666141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.666108    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:01.802201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.802143    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" podUID="0c41af0d-a06e-4304-9972-3179f1ec11b8" containerName="payload-processing" containerID="cri-o://9c789e73cd58f50b034c1b9899228a28f4c294811545c19c90333bfe5e56ffcd" gracePeriod=25
Nov 02 12:36:01.852882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.852450    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:36:01.852882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:01.852510    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 12:36:02.025727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.024784    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerStarted","Data":"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"}
Nov 02 12:36:02.025727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.024880    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerStarted","Data":"1dbd153c3be291b195073dc903e1a6a42f0ae10526f8785f65a61029d7348692"}
Nov 02 12:36:02.052062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.036329    4928 generic.go:356] "Generic (PLEG): container finished" podID="20f333a6-6ea2-4616-a67a-e17ccb28b375" containerID="27a2941f2cb5a636d45f0bdb36a39c43457fc0e6a804d33c94b0397b9ea4fbf5" exitCode=0
Nov 02 12:36:02.052062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.036498    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h" event={"ID":"20f333a6-6ea2-4616-a67a-e17ccb28b375","Type":"ContainerDied","Data":"27a2941f2cb5a636d45f0bdb36a39c43457fc0e6a804d33c94b0397b9ea4fbf5"}
Nov 02 12:36:02.052062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.039241    4928 generic.go:356] "Generic (PLEG): container finished" podID="0c41af0d-a06e-4304-9972-3179f1ec11b8" containerID="9c789e73cd58f50b034c1b9899228a28f4c294811545c19c90333bfe5e56ffcd" exitCode=0
Nov 02 12:36:02.052062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.040021    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" event={"ID":"0c41af0d-a06e-4304-9972-3179f1ec11b8","Type":"ContainerDied","Data":"9c789e73cd58f50b034c1b9899228a28f4c294811545c19c90333bfe5e56ffcd"}
Nov 02 12:36:02.164850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.162589    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podStartSLOduration=2.162572746 podStartE2EDuration="2.162572746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:02.080781554 +0000 UTC m=+559.729835515" watchObservedRunningTime="2026-11-02 12:36:02.162572746 +0000 UTC m=+559.811626703"
Nov 02 12:36:02.183287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.182934    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:36:02.278202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.278165    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\") pod \"0c41af0d-a06e-4304-9972-3179f1ec11b8\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") "
Nov 02 12:36:02.278345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.278279    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\" (UniqueName: \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\") pod \"0c41af0d-a06e-4304-9972-3179f1ec11b8\" (UID: \"0c41af0d-a06e-4304-9972-3179f1ec11b8\") "
Nov 02 12:36:02.278620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.278592    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume" pod "0c41af0d-a06e-4304-9972-3179f1ec11b8" (UID: "0c41af0d-a06e-4304-9972-3179f1ec11b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:02.278745 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.278725    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c41af0d-a06e-4304-9972-3179f1ec11b8-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:02.280986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.280956    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv" pod "0c41af0d-a06e-4304-9972-3179f1ec11b8" (UID: "0c41af0d-a06e-4304-9972-3179f1ec11b8"). InnerVolumeSpecName "kube-api-access-bqnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:02.300146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.300118    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:02.301411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.301390    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:02.301496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.301432    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:02.380005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.379971    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqnkv\" (UniqueName: \"kubernetes.io/projected/0c41af0d-a06e-4304-9972-3179f1ec11b8-kube-api-access-bqnkv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:02.852428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.852338    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:36:02.852428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:02.852399    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 12:36:03.057656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.057625    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"
Nov 02 12:36:03.058144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.057629    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j" event={"ID":"0c41af0d-a06e-4304-9972-3179f1ec11b8","Type":"ContainerDied","Data":"d963a0dedb4580906509ccceecebee1888b0b07cdd141577f17e811d853b78fd"}
Nov 02 12:36:03.058144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.057801    4928 scope.go:122] "RemoveContainer" containerID="9c789e73cd58f50b034c1b9899228a28f4c294811545c19c90333bfe5e56ffcd"
Nov 02 12:36:03.115273 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.115242    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:36:03.121865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.121837    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f080cb5f-69cb9bc795-9sj7j"]
Nov 02 12:36:03.275235 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.275211    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:36:03.304907 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.304881    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:03.395074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.394690    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\") pod \"20f333a6-6ea2-4616-a67a-e17ccb28b375\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") "
Nov 02 12:36:03.395074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.394765    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\" (UniqueName: \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\") pod \"20f333a6-6ea2-4616-a67a-e17ccb28b375\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") "
Nov 02 12:36:03.395074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.394802    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\") pod \"20f333a6-6ea2-4616-a67a-e17ccb28b375\" (UID: \"20f333a6-6ea2-4616-a67a-e17ccb28b375\") "
Nov 02 12:36:03.395815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.395786    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca" pod "20f333a6-6ea2-4616-a67a-e17ccb28b375" (UID: "20f333a6-6ea2-4616-a67a-e17ccb28b375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:03.400040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.399998    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f" pod "20f333a6-6ea2-4616-a67a-e17ccb28b375" (UID: "20f333a6-6ea2-4616-a67a-e17ccb28b375"). InnerVolumeSpecName "kube-api-access-prc2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:03.400219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.400023    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca" pod "20f333a6-6ea2-4616-a67a-e17ccb28b375" (UID: "20f333a6-6ea2-4616-a67a-e17ccb28b375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:36:03.496310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.496271    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:03.496310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.496305    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prc2f\" (UniqueName: \"kubernetes.io/projected/20f333a6-6ea2-4616-a67a-e17ccb28b375-kube-api-access-prc2f\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:03.496310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.496315    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f333a6-6ea2-4616-a67a-e17ccb28b375-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:03.859851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.859182    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:36:03.859851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.859244    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 12:36:03.881868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:03.877800    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:36:04.002783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.002744    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.002957 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.002856    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.002957 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.002931    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.002980    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003031    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003079    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003130    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003234    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003292    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\") pod \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\" (UID: \"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b\") "
Nov 02 12:36:04.003530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003496    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:04.003808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003786    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.003912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.003885    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:04.004045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.004023    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:04.004118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.004041    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:04.004296 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.004269    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:04.007191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.007157    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:04.007493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.007450    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:04.007493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.007452    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "kube-api-access-kzsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:04.007869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.007842    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token" pod "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" (UID: "f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:04.066018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.065979    4928 generic.go:356] "Generic (PLEG): container finished" podID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerID="c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee" exitCode=0
Nov 02 12:36:04.066462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.066037    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerDied","Data":"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"}
Nov 02 12:36:04.066462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.066054    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"
Nov 02 12:36:04.066462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.066107    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc" event={"ID":"f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b","Type":"ContainerDied","Data":"6fcb2db031a17e601e0084fe688e6b12f2040ee715b233b3d7eadd35496cc538"}
Nov 02 12:36:04.066462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.066137    4928 scope.go:122] "RemoveContainer" containerID="c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"
Nov 02 12:36:04.073324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.071795    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"
Nov 02 12:36:04.074005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.073963    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h" event={"ID":"20f333a6-6ea2-4616-a67a-e17ccb28b375","Type":"ContainerDied","Data":"ddb0acf2d7b3320b9384b1304702fffc6572d72e9c16adfa47fc95a2590803c6"}
Nov 02 12:36:04.074174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.074158    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb0acf2d7b3320b9384b1304702fffc6572d72e9c16adfa47fc95a2590803c6"
Nov 02 12:36:04.074280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.074268    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:04.075075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.075054    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:04.080836 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.080784    4928 scope.go:122] "RemoveContainer" containerID="2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.090738    4928 scope.go:122] "RemoveContainer" containerID="c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:04.091026    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee\": container with ID starting with c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee not found: ID does not exist" containerID="c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.091052    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee"} err="failed to get container status \"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee\": rpc error: code = NotFound desc = could not find container \"c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee\": container with ID starting with c701e2ab0e61fe3c9c3448be53b5ded538f86318cc1ae4c771d1f103b04ef4ee not found: ID does not exist"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.091070    4928 scope.go:122] "RemoveContainer" containerID="2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:04.091315    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a\": container with ID starting with 2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a not found: ID does not exist" containerID="2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"
Nov 02 12:36:04.095009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.091334    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a"} err="failed to get container status \"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a\": rpc error: code = NotFound desc = could not find container \"2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a\": container with ID starting with 2b6ccdce940683d5fa955650d403e3697bb3f9191a0b67826057655e39b5232a not found: ID does not exist"
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105504    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105555    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105571    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105587    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105623    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-kube-api-access-kzsvl\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105638    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105653    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.106856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.105671    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:04.160354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.159858    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:36:04.164974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.164945    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f080cb5f-openshift-default-7644d56f7-t5mlc"]
Nov 02 12:36:04.497538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.497504    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c41af0d-a06e-4304-9972-3179f1ec11b8" path="/var/lib/kubelet/pods/0c41af0d-a06e-4304-9972-3179f1ec11b8/volumes"
Nov 02 12:36:04.498000 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.497986    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" path="/var/lib/kubelet/pods/f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b/volumes"
Nov 02 12:36:04.633875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.633822    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:36:04.857329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.857297    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:36:04.921019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:04.920989    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:36:05.013897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.013844    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.013950    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014136    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014185    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014236    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:05.014329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014252    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014429 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014412    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014494 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014471    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014512    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014643 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014598    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\") pod \"aad91c1e-cc09-4114-a851-c5961a98bbc6\" (UID: \"aad91c1e-cc09-4114-a851-c5961a98bbc6\") "
Nov 02 12:36:05.014643 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014613    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:05.014738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014673    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:05.014886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.014865    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:05.015056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.015041    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.015134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.015064    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.015134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.015079    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.015134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.015092    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.015259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.015163    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:05.016758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.016729    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:05.017176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.017153    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:05.017271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.017221    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "kube-api-access-tncxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:05.017271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.017244    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo" pod "aad91c1e-cc09-4114-a851-c5961a98bbc6" (UID: "aad91c1e-cc09-4114-a851-c5961a98bbc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:05.080529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.080433    4928 generic.go:356] "Generic (PLEG): container finished" podID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerID="4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a" exitCode=0
Nov 02 12:36:05.080529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.080502    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"
Nov 02 12:36:05.080529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.080514    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerDied","Data":"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"}
Nov 02 12:36:05.081153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.080605    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx" event={"ID":"aad91c1e-cc09-4114-a851-c5961a98bbc6","Type":"ContainerDied","Data":"0a0ead4e384099d2650c4921134cca18e340982ee4f30c918f8de42fcf834bad"}
Nov 02 12:36:05.081153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.080639    4928 scope.go:122] "RemoveContainer" containerID="4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"
Nov 02 12:36:05.083856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.083801    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:36:05.091004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.090985    4928 scope.go:122] "RemoveContainer" containerID="b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"
Nov 02 12:36:05.102002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.101979    4928 scope.go:122] "RemoveContainer" containerID="4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"
Nov 02 12:36:05.102367 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:05.102344    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a\": container with ID starting with 4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a not found: ID does not exist" containerID="4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"
Nov 02 12:36:05.102458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.102374    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a"} err="failed to get container status \"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a\": rpc error: code = NotFound desc = could not find container \"4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a\": container with ID starting with 4e4219648251d094224b5a1549d998017827193f7caf9ac9efc0a8ab5733558a not found: ID does not exist"
Nov 02 12:36:05.102458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.102394    4928 scope.go:122] "RemoveContainer" containerID="b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"
Nov 02 12:36:05.102669 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:05.102654    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf\": container with ID starting with b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf not found: ID does not exist" containerID="b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"
Nov 02 12:36:05.102706 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.102682    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf"} err="failed to get container status \"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf\": rpc error: code = NotFound desc = could not find container \"b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf\": container with ID starting with b57298496275eabc3f6283378c007b8b3fc9005014f161d42772d42422fe87bf not found: ID does not exist"
Nov 02 12:36:05.116695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.116654    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncxd\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-kube-api-access-tncxd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.116695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.116696    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.116944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.116713    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aad91c1e-cc09-4114-a851-c5961a98bbc6-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.116944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.116727    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.116944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.116741    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aad91c1e-cc09-4114-a851-c5961a98bbc6-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:05.193791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.193753    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:36:05.200168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.200124    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-53b6603d-openshift-default-5b696f6c9f-hsttx"]
Nov 02 12:36:05.807787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.807748    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 12:36:05.808195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808181    4928 cpu_manager.go:414] "containerMap: removing container" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:05.808234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808196    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:05.808234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808222    4928 cpu_manager.go:414] "containerMap: removing container" podUID="20f333a6-6ea2-4616-a67a-e17ccb28b375" containerName="revoke-keys"
Nov 02 12:36:05.808234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808228    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f333a6-6ea2-4616-a67a-e17ccb28b375" containerName="revoke-keys"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808237    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808242    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808249    4928 cpu_manager.go:414] "containerMap: removing container" podUID="0c41af0d-a06e-4304-9972-3179f1ec11b8" containerName="payload-processing"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808254    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c41af0d-a06e-4304-9972-3179f1ec11b8" containerName="payload-processing"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808314    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c41af0d-a06e-4304-9972-3179f1ec11b8" containerName="payload-processing"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808323    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:05.808334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808330    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:05.808530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.808339    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f333a6-6ea2-4616-a67a-e17ccb28b375" containerName="revoke-keys"
Nov 02 12:36:05.812767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.812749    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:05.832912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.832874    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 12:36:05.843910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.843794    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-068756-serving-cert\""
Nov 02 12:36:05.843910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.843845    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:36:05.843910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.843795    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:36:05.843910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.843862    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 12:36:05.846286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.846262    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 12:36:05.924025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.923991    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:05.924227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.924061    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zldpl\" (UniqueName: \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:05.924227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.924203    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:05.924353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:05.924244    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.025280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.025242    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.025461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.025306    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zldpl\" (UniqueName: \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.025461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.025351    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.025461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.025377    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.025934 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.025902    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.027948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.027924    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.027948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.027938    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.052569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.052528    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldpl\" (UniqueName: \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\") pod \"maas-api-e2e-worker-w3-068756-dccdd974-ctf7g\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.126095 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.126006    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:06.341729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.341692    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 12:36:06.342200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342184    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:06.342250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342203    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:06.342250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342230    4928 cpu_manager.go:414] "containerMap: removing container" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:06.342250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342235    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:06.342348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342311    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b7bd7-adeb-4624-8b6f-cf39a4b7ac5b" containerName="istio-proxy"
Nov 02 12:36:06.342348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.342322    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" containerName="istio-proxy"
Nov 02 12:36:06.346777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.346761    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.363065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.363022    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-95ee26-dockercfg-b5b8p\""
Nov 02 12:36:06.363384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.363357    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-95ee26\""
Nov 02 12:36:06.366023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.365998    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 12:36:06.452537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.452043    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 12:36:06.456865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.456154    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9cf\" (UniqueName: \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.456865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.456261    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.467220 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:06.467194    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcaa75b_7486_498d_80eb_97324ac2bdc2.slice/crio-05bf521b65feb5cfd6372d23195dbe1bf3df0179735b0d1e0c5f1ebe4523e7c7 WatchSource:0}: Error finding container 05bf521b65feb5cfd6372d23195dbe1bf3df0179735b0d1e0c5f1ebe4523e7c7: Status 404 returned error can't find the container with id 05bf521b65feb5cfd6372d23195dbe1bf3df0179735b0d1e0c5f1ebe4523e7c7
Nov 02 12:36:06.499876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.499809    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad91c1e-cc09-4114-a851-c5961a98bbc6" path="/var/lib/kubelet/pods/aad91c1e-cc09-4114-a851-c5961a98bbc6/volumes"
Nov 02 12:36:06.557763 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.557716    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.557952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.557877    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9cf\" (UniqueName: \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.558557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.558537    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.582466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.582434    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9cf\" (UniqueName: \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\") pod \"payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.662596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.662558    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:06.975772 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:06.975671    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 12:36:06.979699 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:06.979669    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a98d9e_e77a_436c_923d_3f8a9ae05b15.slice/crio-2f769271270670f69aa53754504bb38205220fbb0f6bccef518ed52d7a2c0b30 WatchSource:0}: Error finding container 2f769271270670f69aa53754504bb38205220fbb0f6bccef518ed52d7a2c0b30: Status 404 returned error can't find the container with id 2f769271270670f69aa53754504bb38205220fbb0f6bccef518ed52d7a2c0b30
Nov 02 12:36:07.110201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.110148    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" event={"ID":"57a98d9e-e77a-436c-923d-3f8a9ae05b15","Type":"ContainerStarted","Data":"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"}
Nov 02 12:36:07.110374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.110244    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" event={"ID":"57a98d9e-e77a-436c-923d-3f8a9ae05b15","Type":"ContainerStarted","Data":"2f769271270670f69aa53754504bb38205220fbb0f6bccef518ed52d7a2c0b30"}
Nov 02 12:36:07.111312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.111290    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:07.114316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.114285    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" event={"ID":"cdcaa75b-7486-498d-80eb-97324ac2bdc2","Type":"ContainerStarted","Data":"05bf521b65feb5cfd6372d23195dbe1bf3df0179735b0d1e0c5f1ebe4523e7c7"}
Nov 02 12:36:07.161073 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.160861    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" podStartSLOduration=1.1608409 podStartE2EDuration="1.1608409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:07.157423199 +0000 UTC m=+564.806477151" watchObservedRunningTime="2026-11-02 12:36:07.1608409 +0000 UTC m=+564.809894857"
Nov 02 12:36:07.307922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.307876    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 12:36:07.312446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.312418    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.333448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.333396    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843978-dockercfg-vt7p6\""
Nov 02 12:36:07.334004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.333671    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843978\""
Nov 02 12:36:07.345039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.345001    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 12:36:07.428923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.428876    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 12:36:07.436439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.436408    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.439464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.439417    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 12:36:07.473364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.473330    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqjh\" (UniqueName: \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.473538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.473378    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.575535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.574345    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqjh\" (UniqueName: \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.575535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.574407    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.575535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.574490    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j556x\" (UniqueName: \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.575535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.574540    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.575909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.575656    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.599943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.599907    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqjh\" (UniqueName: \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\") pod \"payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.644062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.644024    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:07.675801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.675761    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j556x\" (UniqueName: \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.675932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.675862    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.677167 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.676788    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.704532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.704278    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j556x\" (UniqueName: \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\") pod \"payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.762268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.762230    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:07.976434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:07.976402    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 12:36:07.978679 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:07.978644    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ebad1b_0ba4_4ec1_959e_8e1e9f1b9a25.slice/crio-aba31ac645fe418d2e16a876f3f79a2eacaeeb9d8cc1293fd297f2cf31496d8e WatchSource:0}: Error finding container aba31ac645fe418d2e16a876f3f79a2eacaeeb9d8cc1293fd297f2cf31496d8e: Status 404 returned error can't find the container with id aba31ac645fe418d2e16a876f3f79a2eacaeeb9d8cc1293fd297f2cf31496d8e
Nov 02 12:36:08.073210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.073180    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 12:36:08.075443 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:08.075410    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a9f87a_1272_4591_bd5d_df59619b1f28.slice/crio-ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100 WatchSource:0}: Error finding container ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100: Status 404 returned error can't find the container with id ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100
Nov 02 12:36:08.125390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.125338    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" event={"ID":"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25","Type":"ContainerStarted","Data":"4882cca20dc362d6b02bc1ef2b7ca4ace978c233fafc9fc40cc79139c79bb27a"}
Nov 02 12:36:08.125555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.125402    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" event={"ID":"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25","Type":"ContainerStarted","Data":"aba31ac645fe418d2e16a876f3f79a2eacaeeb9d8cc1293fd297f2cf31496d8e"}
Nov 02 12:36:08.125555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.125523    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:08.128341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.128240    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" event={"ID":"14a9f87a-1272-4591-bd5d-df59619b1f28","Type":"ContainerStarted","Data":"ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100"}
Nov 02 12:36:08.176450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:08.176398    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" podStartSLOduration=1.176384629 podStartE2EDuration="1.176384629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:08.17415491 +0000 UTC m=+565.823208869" watchObservedRunningTime="2026-11-02 12:36:08.176384629 +0000 UTC m=+565.825438588"
Nov 02 12:36:09.139910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:09.139874    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" event={"ID":"14a9f87a-1272-4591-bd5d-df59619b1f28","Type":"ContainerStarted","Data":"2044d0409ce35fb2ed2eb81112a1830203dc948b77652c6f91c4febb7b7f681c"}
Nov 02 12:36:09.140044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:09.139938    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:09.183147 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:09.183090    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" podStartSLOduration=2.18307471 podStartE2EDuration="2.18307471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:09.181589577 +0000 UTC m=+566.830643535" watchObservedRunningTime="2026-11-02 12:36:09.18307471 +0000 UTC m=+566.832128668"
Nov 02 12:36:09.728658 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:09.728617    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:36:09.744225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:09.744194    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"]
Nov 02 12:36:09.757787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:09.757754    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-628cf9bd-kl8tl"]
Nov 02 12:36:10.148519 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:10.148477    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" event={"ID":"cdcaa75b-7486-498d-80eb-97324ac2bdc2","Type":"ContainerStarted","Data":"693236a15e400921a03bd3a76d9c52859e682f6da2ce6096f579e402386e42ee"}
Nov 02 12:36:10.196001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:10.195934    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" podStartSLOduration=2.5931091950000003 podStartE2EDuration="5.195917047s" totalImagesPullingTime="2.602807852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:10.194169278 +0000 UTC m=+567.843223236" watchObservedRunningTime="2026-11-02 12:36:10.195917047 +0000 UTC m=+567.844970983"
Nov 02 12:36:10.504506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:10.504470    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789821d8-c3e9-4093-8ae0-31b286fb452d" path="/var/lib/kubelet/pods/789821d8-c3e9-4093-8ae0-31b286fb452d/volumes"
Nov 02 12:36:11.172362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.172332    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:11.653899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.653799    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:36:11.658848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.658801    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.679268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.679239    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:36:11.689132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.689110    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5514e5-openshift-default-dockercfg-7v87s\""
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829398    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829478    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829529    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829558    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96csm\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829586    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829642    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829683    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829739    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.844848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.829770    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933419    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933504    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933554    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933578    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96csm\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933613    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933658    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933706    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933768    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.933799    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.934718    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.935896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.935333    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.936515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.936164    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.941077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.941049    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.941494 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.941475    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.941881 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.941863    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.948908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.946365    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.965905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.964382    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.965905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.964964    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96csm\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:11.976324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:11.976290    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:12.355860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:12.354462    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:36:12.356337 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:12.356302    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318fa227_ec90_4c1c_8870_21803b84b28f.slice/crio-210b59618bc77806b0acce961471f7768521c7029af3966cb76683270ddac150 WatchSource:0}: Error finding container 210b59618bc77806b0acce961471f7768521c7029af3966cb76683270ddac150: Status 404 returned error can't find the container with id 210b59618bc77806b0acce961471f7768521c7029af3966cb76683270ddac150
Nov 02 12:36:12.359819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:12.359777    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:12.359948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:12.359890    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:12.359948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:12.359929    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:13.187447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:13.187410    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerStarted","Data":"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"}
Nov 02 12:36:13.187950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:13.187477    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerStarted","Data":"210b59618bc77806b0acce961471f7768521c7029af3966cb76683270ddac150"}
Nov 02 12:36:13.977123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:13.977074    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:13.982640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:13.982613    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:14.041451 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:14.041381    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podStartSLOduration=3.041362414 podStartE2EDuration="3.041362414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:13.244182547 +0000 UTC m=+570.893236508" watchObservedRunningTime="2026-11-02 12:36:14.041362414 +0000 UTC m=+571.690416373"
Nov 02 12:36:14.193488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:14.193450    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:14.194443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:14.194421    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:17.185612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:17.185529    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 12:36:17.410589 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:17.410552    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"]
Nov 02 12:36:17.451489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:17.451390    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f0bef09-wtw2h"]
Nov 02 12:36:17.576820 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:17.576785    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:36:17.577406 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:17.577360    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" containerID="cri-o://91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813" gracePeriod=30
Nov 02 12:36:18.498137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:18.498097    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f333a6-6ea2-4616-a67a-e17ccb28b375" path="/var/lib/kubelet/pods/20f333a6-6ea2-4616-a67a-e17ccb28b375/volumes"
Nov 02 12:36:19.154973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:19.153943    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 12:36:19.155173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:19.154978    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 12:36:20.085336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:20.085296    4928 patch_prober.go:29] interesting pod/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:36:20.085796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:20.085358    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:36:20.162311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:20.162266    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 12:36:22.293260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.293226    4928 generic.go:356] "Generic (PLEG): container finished" podID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerID="1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1" exitCode=0
Nov 02 12:36:22.293765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.293303    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerDied","Data":"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"}
Nov 02 12:36:22.293868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.293768    4928 scope.go:122] "RemoveContainer" containerID="1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"
Nov 02 12:36:22.295195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.295164    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:22.295294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.295254    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:22.295360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.295304    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:22.909986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:22.909956    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:36:23.072281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072246    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072297    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072337    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072355    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072410    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072690 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072571    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072690 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072679    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072718    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.072793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072725    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:23.072793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072739    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:23.072793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072774    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\") pod \"40a94e41-147d-4ac0-907d-45fc6480e232\" (UID: \"40a94e41-147d-4ac0-907d-45fc6480e232\") "
Nov 02 12:36:23.073022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072936    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:23.073022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.072938    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:23.073136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073047    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:23.073287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073259    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.073425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073287    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.073425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073303    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.073425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073317    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.073425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.073333    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a94e41-147d-4ac0-907d-45fc6480e232-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.074951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.074865    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:23.074951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.074873    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:23.075090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.075044    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:23.075228 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.075208    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8" pod "40a94e41-147d-4ac0-907d-45fc6480e232" (UID: "40a94e41-147d-4ac0-907d-45fc6480e232"). InnerVolumeSpecName "kube-api-access-gvww8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:23.174798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.174698    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a94e41-147d-4ac0-907d-45fc6480e232-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.174798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.174733    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvww8\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-kube-api-access-gvww8\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.174798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.174744    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a94e41-147d-4ac0-907d-45fc6480e232-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.174798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.174753    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a94e41-147d-4ac0-907d-45fc6480e232-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:36:23.303868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.303818    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerStarted","Data":"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"}
Nov 02 12:36:23.305667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.305640    4928 generic.go:356] "Generic (PLEG): container finished" podID="40a94e41-147d-4ac0-907d-45fc6480e232" containerID="91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813" exitCode=0
Nov 02 12:36:23.305786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.305674    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerDied","Data":"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"}
Nov 02 12:36:23.305786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.305702    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"
Nov 02 12:36:23.305786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.305722    4928 scope.go:122] "RemoveContainer" containerID="91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"
Nov 02 12:36:23.305942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.305711    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c" event={"ID":"40a94e41-147d-4ac0-907d-45fc6480e232","Type":"ContainerDied","Data":"06fb5e9a783c9707706521ede005a704f2c75d4dfc2996ca92d260a87b181536"}
Nov 02 12:36:23.315790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.315765    4928 scope.go:122] "RemoveContainer" containerID="21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"
Nov 02 12:36:23.354151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.354103    4928 scope.go:122] "RemoveContainer" containerID="91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"
Nov 02 12:36:23.355231 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:23.355122    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813\": container with ID starting with 91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813 not found: ID does not exist" containerID="91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"
Nov 02 12:36:23.355231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.355179    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813"} err="failed to get container status \"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813\": rpc error: code = NotFound desc = could not find container \"91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813\": container with ID starting with 91feeb609d01079ea70aacdb0181e90ca147e602cee1a075eca9d9fcdedda813 not found: ID does not exist"
Nov 02 12:36:23.355503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.355411    4928 scope.go:122] "RemoveContainer" containerID="21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"
Nov 02 12:36:23.356018 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:23.355963    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5\": container with ID starting with 21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5 not found: ID does not exist" containerID="21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"
Nov 02 12:36:23.356018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.355994    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5"} err="failed to get container status \"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5\": rpc error: code = NotFound desc = could not find container \"21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5\": container with ID starting with 21bd62e283674507eadd6f7a68db3946fa0ab34211285046925a29636a16e0a5 not found: ID does not exist"
Nov 02 12:36:23.421013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.420966    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:36:23.424785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.424756    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2f0bef09-openshift-default-6d9ff87485-mrq7c"]
Nov 02 12:36:23.553860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.552199    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:23.553860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.552327    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:36:23.553860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:23.552371    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 12:36:24.498006 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:24.497975    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" path="/var/lib/kubelet/pods/40a94e41-147d-4ac0-907d-45fc6480e232/volumes"
Nov 02 12:36:24.507029 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:24.507001    4928 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@5-14243-0.service\": RecentStats: unable to find data in memory cache]"
Nov 02 12:36:24.556247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:24.556202    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:36:24.556462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:24.556280    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 12:36:25.323231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.323195    4928 generic.go:356] "Generic (PLEG): container finished" podID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerID="6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4" exitCode=0
Nov 02 12:36:25.323443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.323253    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerDied","Data":"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"}
Nov 02 12:36:25.323714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.323689    4928 scope.go:122] "RemoveContainer" containerID="6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"
Nov 02 12:36:25.351861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.340153    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:25.351861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.340275    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:25.351861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.340321    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:25.563855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.562315    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:36:25.563855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.562401    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 12:36:25.856857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.856157    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 12:36:25.857086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857044    4928 cpu_manager.go:414] "containerMap: removing container" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.857086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857069    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.857232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857101    4928 cpu_manager.go:414] "containerMap: removing container" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.857232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857110    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.857353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857255    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.857353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.857278    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a94e41-147d-4ac0-907d-45fc6480e232" containerName="istio-proxy"
Nov 02 12:36:25.870065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.864675    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:25.870065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.869562    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 12:36:25.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.889481    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-61e341\""
Nov 02 12:36:25.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.890279    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-61e341-dockercfg-lz476\""
Nov 02 12:36:25.973989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.973378    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:25.973989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.973504    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:25.982782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.982434    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 12:36:25.994087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:25.987879    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.002610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.002565    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 12:36:26.079461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.079420    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:26.079767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.079742    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:26.079951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.079929    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5ll\" (UniqueName: \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.080183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.080165    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.081839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.081351    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:26.124729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.122168    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\") pod \"payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:26.183850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.182950    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5ll\" (UniqueName: \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.183850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.183098    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.186852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.184122    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.208721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.208691    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5ll\" (UniqueName: \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\") pod \"payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.291333 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.290896    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:26.315264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.315235    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:26.404198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.404161    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerStarted","Data":"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"}
Nov 02 12:36:26.554865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.554326    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:36:26.554865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.554414    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 12:36:26.680866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.680743    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 12:36:26.684598 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:26.684559    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ceb4ec_4e33_42be_81d6_e307fdb70ee9.slice/crio-395386ca730610849b2ad252b7d706b210f2b80f71963ce8b8c0a8d52ec304e1 WatchSource:0}: Error finding container 395386ca730610849b2ad252b7d706b210f2b80f71963ce8b8c0a8d52ec304e1: Status 404 returned error can't find the container with id 395386ca730610849b2ad252b7d706b210f2b80f71963ce8b8c0a8d52ec304e1
Nov 02 12:36:26.779850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:26.778764    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 12:36:26.801121 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:26.801079    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5186cc_caf9_462b_9022_adf788b4ecec.slice/crio-0aea8766aed93407182e3245bf4998686efd202c0c577b9864731149bdfeaafa WatchSource:0}: Error finding container 0aea8766aed93407182e3245bf4998686efd202c0c577b9864731149bdfeaafa: Status 404 returned error can't find the container with id 0aea8766aed93407182e3245bf4998686efd202c0c577b9864731149bdfeaafa
Nov 02 12:36:27.300072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.300041    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:27.300283 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.300133    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:27.300283 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.300165    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:27.413448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.413412    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" event={"ID":"12ceb4ec-4e33-42be-81d6-e307fdb70ee9","Type":"ContainerStarted","Data":"30b7132eb185b3984a96b355d3131c14bb8f9e6a8442893f6909f99c247ef07d"}
Nov 02 12:36:27.413622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.413468    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" event={"ID":"12ceb4ec-4e33-42be-81d6-e307fdb70ee9","Type":"ContainerStarted","Data":"395386ca730610849b2ad252b7d706b210f2b80f71963ce8b8c0a8d52ec304e1"}
Nov 02 12:36:27.414160 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.414144    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:27.416018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.415993    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" event={"ID":"8a5186cc-caf9-462b-9022-adf788b4ecec","Type":"ContainerStarted","Data":"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"}
Nov 02 12:36:27.416114 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.416034    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" event={"ID":"8a5186cc-caf9-462b-9022-adf788b4ecec","Type":"ContainerStarted","Data":"0aea8766aed93407182e3245bf4998686efd202c0c577b9864731149bdfeaafa"}
Nov 02 12:36:27.518837 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.518770    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" podStartSLOduration=2.518754887 podStartE2EDuration="2.518754887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:27.516959855 +0000 UTC m=+585.166013813" watchObservedRunningTime="2026-11-02 12:36:27.518754887 +0000 UTC m=+585.167808846"
Nov 02 12:36:27.519772 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.519733    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" podStartSLOduration=2.51972429 podStartE2EDuration="2.51972429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:27.474899676 +0000 UTC m=+585.123953637" watchObservedRunningTime="2026-11-02 12:36:27.51972429 +0000 UTC m=+585.168778241"
Nov 02 12:36:27.553312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.553215    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:36:27.553579 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:27.553546    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 12:36:28.300434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.300396    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:28.300887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.300454    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:28.423497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.423452    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:28.893063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.891361    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:28.893063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.892911    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:28.897021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:28.896997    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:36:29.300558 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:29.300526    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:29.300980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:29.300579    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:30.300748 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:30.300702    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:30.301198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:30.300767    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:31.203072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.203040    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 12:36:31.209912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.209890    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.247860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.236078    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-61e341-serving-cert\""
Nov 02 12:36:31.275930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.275879    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 12:36:31.300526 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.300481    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:36:31.300804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.300768    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 02 12:36:31.306820 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.306783    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.307078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.307058    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.307294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.307278    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjkp\" (UniqueName: \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.307413 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.307400    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.408562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.408526    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.408755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.408608    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.408755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.408695    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjkp\" (UniqueName: \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.408755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.408722    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.409641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.409613    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.411129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.411103    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.411391 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.411370    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.447695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.447664    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjkp\" (UniqueName: \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\") pod \"maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.558561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.558524    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:31.878275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:31.878242    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 12:36:31.880047 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:31.880011    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fcc9b0_cf30_4e1a_99b7_b289f329e832.slice/crio-694b3d0a4eafa62e4a77261f242e88577674bd87c4a8b032c49602090285e466 WatchSource:0}: Error finding container 694b3d0a4eafa62e4a77261f242e88577674bd87c4a8b032c49602090285e466: Status 404 returned error can't find the container with id 694b3d0a4eafa62e4a77261f242e88577674bd87c4a8b032c49602090285e466
Nov 02 12:36:32.304258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.304227    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:32.304705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.304506    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:32.305211 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.305190    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:36:32.512892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.512849    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" event={"ID":"b0fcc9b0-cf30-4e1a-99b7-b289f329e832","Type":"ContainerStarted","Data":"ea4d12d3d8e41b83915c421711f2ec30e62a6e9507deae0b568c0f7001e38b5c"}
Nov 02 12:36:32.513085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.512913    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" event={"ID":"b0fcc9b0-cf30-4e1a-99b7-b289f329e832","Type":"ContainerStarted","Data":"694b3d0a4eafa62e4a77261f242e88577674bd87c4a8b032c49602090285e466"}
Nov 02 12:36:32.513085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.512965    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:32.560179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:32.560073    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" podStartSLOduration=1.086500604 podStartE2EDuration="1.560052279s" totalImagesPullingTime="473.551675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:32.558996767 +0000 UTC m=+590.208050727" watchObservedRunningTime="2026-11-02 12:36:32.560052279 +0000 UTC m=+590.209106237"
Nov 02 12:36:38.525204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.525116    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 12:36:38.580630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.580590    4928 generic.go:356] "Generic (PLEG): container finished" podID="318fa227-ec90-4c1c-8870-21803b84b28f" containerID="28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b" exitCode=0
Nov 02 12:36:38.580810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.580671    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerDied","Data":"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"}
Nov 02 12:36:38.581233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.581209    4928 scope.go:122] "RemoveContainer" containerID="28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"
Nov 02 12:36:38.583433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.583393    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:38.583536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.583481    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:38.583536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:38.583526    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:36:39.471902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.471856    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 12:36:39.472089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.471988    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 12:36:39.590495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.590456    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerStarted","Data":"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"}
Nov 02 12:36:39.770104 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:36:39.770066    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:36:39.977499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.977452    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:39.979029 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.978997    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:36:39.979170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:39.979063    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 12:36:40.977387 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:40.977351    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:36:40.977806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:40.977411    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 12:36:41.977166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:41.977129    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:36:41.977505 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:41.977190    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 12:36:42.607443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:42.607414    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:36:42.613380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:42.613359    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:36:42.977685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:42.977576    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:36:42.977685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:42.977640    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 12:36:43.977773 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:43.977739    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:36:43.978225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:43.977799    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 12:36:43.987136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:43.987104    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 12:36:43.993431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:43.993408    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.009936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.009906    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 12:36:44.011549 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.011520    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.011806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.011784    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.012025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.012002    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.012130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.012032    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5k7s\" (UniqueName: \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.012200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.012145    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.012200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.012173    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.020208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.020185    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"kube-root-ca.crt\""
Nov 02 12:36:44.020325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.020185    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"openshift-service-ca.crt\""
Nov 02 12:36:44.020325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.020232    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"default-dockercfg-vbmk6\""
Nov 02 12:36:44.023017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.022990    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"simulator-w0-5514e5-kserve-self-signed-certs\""
Nov 02 12:36:44.112911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.112875    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.112934    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.112968    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5k7s\" (UniqueName: \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113115    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113158    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113196    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113421    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113519    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.113613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.113585    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.115479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.115447    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.115606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.115584    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.152695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.152631    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5k7s\" (UniqueName: \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\") pod \"simulator-w0-5514e5-kserve-548559875b-j2fxw\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") " pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.194135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.194102    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:44.306265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.306226    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:44.642078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.642045    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 12:36:44.644035 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:44.644007    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb351b1bd_4af8_4234_bd41_fa92e753e8ac.slice/crio-02bfc8b65eb3fd25b53d379f9cf0c5ef73db2efa1b3ea0d6a565e183095e4b0d WatchSource:0}: Error finding container 02bfc8b65eb3fd25b53d379f9cf0c5ef73db2efa1b3ea0d6a565e183095e4b0d: Status 404 returned error can't find the container with id 02bfc8b65eb3fd25b53d379f9cf0c5ef73db2efa1b3ea0d6a565e183095e4b0d
Nov 02 12:36:44.981503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.981423    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:44.982611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:44.982589    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:36:45.630596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:45.630542    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" event={"ID":"b351b1bd-4af8-4234-bd41-fa92e753e8ac","Type":"ContainerStarted","Data":"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"}
Nov 02 12:36:45.630819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:45.630610    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" event={"ID":"b351b1bd-4af8-4234-bd41-fa92e753e8ac","Type":"ContainerStarted","Data":"02bfc8b65eb3fd25b53d379f9cf0c5ef73db2efa1b3ea0d6a565e183095e4b0d"}
Nov 02 12:36:45.677212 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:45.677158    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podStartSLOduration=2.677144985 podStartE2EDuration="2.677144985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:45.674522963 +0000 UTC m=+603.323576922" watchObservedRunningTime="2026-11-02 12:36:45.677144985 +0000 UTC m=+603.326198996"
Nov 02 12:36:46.306347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:46.306312    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:54.306630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:54.306589    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:54.320026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:54.320000    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:54.707758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:54.707684    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:36:56.836364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.836322    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 12:36:56.841711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.841680    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.862187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.862153    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 12:36:56.871082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.871047    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"default-dockercfg-bhwfk\""
Nov 02 12:36:56.871082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.871047    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"kube-root-ca.crt\""
Nov 02 12:36:56.875215 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.875183    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"openshift-service-ca.crt\""
Nov 02 12:36:56.875366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.875271    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"simulator-w1-61e341-kserve-self-signed-certs\""
Nov 02 12:36:56.955762 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.955719    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.955953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.955787    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.955953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.955880    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.956080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.955970    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.956080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.956069    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n867j\" (UniqueName: \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:56.956172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:56.956117    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.057695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.057656    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.057897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.057724    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n867j\" (UniqueName: \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.057897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.057768    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.057897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.057860    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.058077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.057908    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.058077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.058043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.058210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.058184    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.058370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.058340    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.058502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.058453    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.060702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.060664    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.065201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.065137    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.100742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.100670    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n867j\" (UniqueName: \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\") pod \"simulator-w1-61e341-kserve-6c9fb87686-mcbqj\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") " pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.157785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.157750    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:36:57.471562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.471535    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 12:36:57.473406 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:36:57.473378    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9206a4db_80cd_45f5_b047_b130365277eb.slice/crio-040fadbdc31575a31a76a76a976c7bb2384a80d32db8e0e0a7f01bae1ce195c4 WatchSource:0}: Error finding container 040fadbdc31575a31a76a76a976c7bb2384a80d32db8e0e0a7f01bae1ce195c4: Status 404 returned error can't find the container with id 040fadbdc31575a31a76a76a976c7bb2384a80d32db8e0e0a7f01bae1ce195c4
Nov 02 12:36:57.717549 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.717512    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" event={"ID":"9206a4db-80cd-45f5-b047-b130365277eb","Type":"ContainerStarted","Data":"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"}
Nov 02 12:36:57.717724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.717564    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" event={"ID":"9206a4db-80cd-45f5-b047-b130365277eb","Type":"ContainerStarted","Data":"040fadbdc31575a31a76a76a976c7bb2384a80d32db8e0e0a7f01bae1ce195c4"}
Nov 02 12:36:57.766031 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:57.765979    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podStartSLOduration=1.765964831 podStartE2EDuration="1.765964831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:57.764391177 +0000 UTC m=+615.413445136" watchObservedRunningTime="2026-11-02 12:36:57.765964831 +0000 UTC m=+615.415018789"
Nov 02 12:36:58.158375 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:36:58.158284    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:37:07.158182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:07.158147    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:37:07.171693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:07.171655    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:37:07.798891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:07.798857    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:37:08.224497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.224401    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 12:37:08.229771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.229744    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.247937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.247906    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 12:37:08.258057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.258020    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"kube-root-ca.crt\""
Nov 02 12:37:08.258057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.258029    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"default-dockercfg-99n5t\""
Nov 02 12:37:08.259275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.259255    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"openshift-service-ca.crt\""
Nov 02 12:37:08.259338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.259307    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"premium-w3-068756-kserve-self-signed-certs\""
Nov 02 12:37:08.382712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.382664    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.382923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.382747    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.382989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.382910    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.383043 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.383021    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.383080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.383054    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.383188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.383169    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bsf\" (UniqueName: \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.484743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.484706    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.484979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.484777    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.484979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.484800    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.484979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.484880    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bsf\" (UniqueName: \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.484979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.484959    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.485197 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.485023    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.485360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.485327    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.485360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.485354    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.485521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.485429    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.487202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.487180    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.487505 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.487476    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.509692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.509662    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bsf\" (UniqueName: \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\") pod \"premium-w3-068756-kserve-7f75685cc9-hqpkw\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") " pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.545400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.545333    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:08.856308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:08.856277    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 12:37:08.858113 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:37:08.858083    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6357a7d_7a1a_4a03_a008_e0e97b6413d1.slice/crio-d874bf7739f7545c7417d70e1eb67b03dd842f9fd5e1aba3223e50458b3eac00 WatchSource:0}: Error finding container d874bf7739f7545c7417d70e1eb67b03dd842f9fd5e1aba3223e50458b3eac00: Status 404 returned error can't find the container with id d874bf7739f7545c7417d70e1eb67b03dd842f9fd5e1aba3223e50458b3eac00
Nov 02 12:37:09.728575 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:37:09.728533    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:37:09.803403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:09.803359    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" event={"ID":"d6357a7d-7a1a-4a03-a008-e0e97b6413d1","Type":"ContainerStarted","Data":"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"}
Nov 02 12:37:09.803403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:09.803408    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" event={"ID":"d6357a7d-7a1a-4a03-a008-e0e97b6413d1","Type":"ContainerStarted","Data":"d874bf7739f7545c7417d70e1eb67b03dd842f9fd5e1aba3223e50458b3eac00"}
Nov 02 12:37:09.887841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:09.887771    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podStartSLOduration=1.887757009 podStartE2EDuration="1.887757009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:09.885183034 +0000 UTC m=+627.534236992" watchObservedRunningTime="2026-11-02 12:37:09.887757009 +0000 UTC m=+627.536810967"
Nov 02 12:37:10.546445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:10.546412    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:18.546137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:18.546094    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:18.558726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:18.558696    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:18.877771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:18.877681    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:37:30.192959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.192924    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 12:37:30.196968 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.196944    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.214002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.213980    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-95ee26-openshift-default-dockercfg-xjh8d\""
Nov 02 12:37:30.233843 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.230651    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 12:37:30.250368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250334    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250385    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250500    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250638 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250550    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250698 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250675    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250718    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnt4q\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.250896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250868    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.251027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250922    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.251027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.250965    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351639    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351683    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351801    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351880    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351941    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.351998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.351968    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt4q\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352052    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352105    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352149    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352151    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352484    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352576    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352597    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.352725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.352648    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.354417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.354388    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.354556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.354451    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.377072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.377029    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt4q\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.377221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.377191    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.510415 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.510381    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:30.839167 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.839140    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 12:37:30.840944 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:37:30.840917    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88845a7d_45bb_42e0_ba81_1c17f3791c95.slice/crio-13a077d70267dddb1a2f72ec4bc3fc9af84130b60a8976df7ba94a4248244cf9 WatchSource:0}: Error finding container 13a077d70267dddb1a2f72ec4bc3fc9af84130b60a8976df7ba94a4248244cf9: Status 404 returned error can't find the container with id 13a077d70267dddb1a2f72ec4bc3fc9af84130b60a8976df7ba94a4248244cf9
Nov 02 12:37:30.843182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.843149    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:37:30.843244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.843211    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:37:30.843280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.843242    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:37:30.948622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.948583    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" event={"ID":"88845a7d-45bb-42e0-ba81-1c17f3791c95","Type":"ContainerStarted","Data":"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"}
Nov 02 12:37:30.948818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.948638    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" event={"ID":"88845a7d-45bb-42e0-ba81-1c17f3791c95","Type":"ContainerStarted","Data":"13a077d70267dddb1a2f72ec4bc3fc9af84130b60a8976df7ba94a4248244cf9"}
Nov 02 12:37:30.997518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:30.997459    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podStartSLOduration=0.99744468 podStartE2EDuration="997.44468ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:30.996373918 +0000 UTC m=+648.645427876" watchObservedRunningTime="2026-11-02 12:37:30.99744468 +0000 UTC m=+648.646498637"
Nov 02 12:37:31.511336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:31.511306    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:31.513019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:31.512991    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:31.513124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:31.513043    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:32.511732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:32.511699    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:32.512176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:32.511748    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:33.510874 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:33.510817    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:33.511049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:33.510890    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:34.511765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:34.511715    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:34.512186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:34.511794    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:35.511359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:35.511315    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:35.511569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:35.511383    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:36.511667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:36.511631    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 02 12:37:36.512084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:36.511697    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 02 12:37:37.515168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:37.515131    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:37.515652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:37.515493    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:37.516163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:37:37.516143    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 12:37:39.729895 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:37:39.729861    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:38:09.729155 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:38:09.729070    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:38:25.651557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.651517    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:38:25.657448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.657403    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.673411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.673387    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:38:25.683570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.683540    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"e2e-conflict-a-7590c91f-kserve-self-signed-certs\""
Nov 02 12:38:25.717438 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717413    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.717551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717460    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.717551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717530    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpssw\" (UniqueName: \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.717684 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717667    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.717769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717710    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.717769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.717738    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819178    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819224    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpssw\" (UniqueName: \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819372    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819398    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.819891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.819853    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.820133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.820102    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.820307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.820132    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.820307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.820205    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.823321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.823287    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.851129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.837121    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.857579 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.857544    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpssw\" (UniqueName: \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\") pod \"e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:25.971621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:25.971531    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:26.309878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:26.305843    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:38:26.370113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:26.370081    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" event={"ID":"7a38229e-4c15-4b5c-a53c-a60d344f0ca4","Type":"ContainerStarted","Data":"a1a452d3678aa368c873fc013970ea78cc2769d64be53b6944f8fe8ec06d20fc"}
Nov 02 12:38:27.390462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:27.390421    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" event={"ID":"7a38229e-4c15-4b5c-a53c-a60d344f0ca4","Type":"ContainerStarted","Data":"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"}
Nov 02 12:38:27.434745 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:27.434682    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" podStartSLOduration=2.434665584 podStartE2EDuration="2.434665584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:27.433651607 +0000 UTC m=+705.082705573" watchObservedRunningTime="2026-11-02 12:38:27.434665584 +0000 UTC m=+705.083719542"
Nov 02 12:38:27.974850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:27.973957    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:31.251202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.251164    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:38:31.255229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.255209    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.276982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.276953    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:38:31.286989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.286959    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 12:38:31.287168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287030    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 12:38:31.287168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287090    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 12:38:31.287441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287423    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 12:38:31.287528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287509    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5ef6342599ae\""
Nov 02 12:38:31.287602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287512    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1944a056069f\""
Nov 02 12:38:31.287602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287512    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee1f245905a7\""
Nov 02 12:38:31.287602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287558    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ca6e84d056b\""
Nov 02 12:38:31.287765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.287612    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-242ww\""
Nov 02 12:38:31.367621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.367590    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.367777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.367632    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.367777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.367761    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.367933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.367783    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78q9\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.469334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.469297    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.469334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.469339    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.469628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.469419    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.469628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.469451    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78q9\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.470095 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.470063    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.470315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.470294    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.472317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.472295    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.494648 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.494616    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78q9\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\") pod \"payload-processing-external-model-84545c695-chvlt\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") " pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.568554 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.568469    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:31.875252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:31.875223    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:38:31.876238 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:38:31.876209    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90983006_b8fe_4a63_b6ad_1808ba505438.slice/crio-87461891444df9bf275e419b53f5df3ffb5453725b8ad3aeb26c2ad9ccd351d9 WatchSource:0}: Error finding container 87461891444df9bf275e419b53f5df3ffb5453725b8ad3aeb26c2ad9ccd351d9: Status 404 returned error can't find the container with id 87461891444df9bf275e419b53f5df3ffb5453725b8ad3aeb26c2ad9ccd351d9
Nov 02 12:38:32.450219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:32.450176    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" event={"ID":"90983006-b8fe-4a63-b6ad-1808ba505438","Type":"ContainerStarted","Data":"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"}
Nov 02 12:38:32.450657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:32.450238    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" event={"ID":"90983006-b8fe-4a63-b6ad-1808ba505438","Type":"ContainerStarted","Data":"87461891444df9bf275e419b53f5df3ffb5453725b8ad3aeb26c2ad9ccd351d9"}
Nov 02 12:38:32.450657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:32.450302    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:32.502516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:32.502451    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" podStartSLOduration=1.502430374 podStartE2EDuration="1.502430374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:32.501867315 +0000 UTC m=+710.150921271" watchObservedRunningTime="2026-11-02 12:38:32.502430374 +0000 UTC m=+710.151484386"
Nov 02 12:38:35.972473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:35.972431    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:35.986927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:35.986898    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:36.487609 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:36.487576    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:38:39.814743 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:38:39.814685    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:38:43.461206 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:43.461174    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:38:45.793184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.793132    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 12:38:45.799913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.799887    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.833568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.830026    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"distinct-w0-5514e5-kserve-self-signed-certs\""
Nov 02 12:38:45.906632 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.906588    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.906964 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.906939    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.907181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.907149    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.936851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.907341    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9wv\" (UniqueName: \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.939518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.939489    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.939770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.939739    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:45.991679 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:45.991641    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 12:38:46.046645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.046545    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.046986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.046966    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.047145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.047129    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.047297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.047273    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9wv\" (UniqueName: \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.047424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.047410    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.047543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.047530    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.048231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.048202    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.049353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.049329    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.050124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.050105    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.081695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.081661    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.105362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.105323    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.115548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.108524    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9wv\" (UniqueName: \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\") pod \"distinct-w0-5514e5-kserve-5ddc4c7788-m82wz\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.301139 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.301018    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:46.681850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.662879    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:38:46.722122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.721075    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:38:46.722122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.721261    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.914574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914477    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.914574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914564    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914610    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914650    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914680    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914745    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4std\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914813    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914858    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:46.915108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:46.914972    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.183314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.172498    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4std\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.183314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.172575    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.183314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.172632    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.183314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.172729    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.183314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.172767    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.186894    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.186998    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.187049    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.187082    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.187611    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.204811    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.205218    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.205499    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.215130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.206629    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.246131 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.222468    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.278851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.271650    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.278851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.271743    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.407654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.407588    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.408504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.408473    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4std\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.478970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.478937    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:47.691796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.690136    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" event={"ID":"e9e0a2e6-1818-48a5-84c2-1d69cb88784d","Type":"ContainerStarted","Data":"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"}
Nov 02 12:38:47.691796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.690190    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" event={"ID":"e9e0a2e6-1818-48a5-84c2-1d69cb88784d","Type":"ContainerStarted","Data":"d446ee593b993fdf16f6bbae296c63409f5ed9e9d79f64d5c998cccd942402e3"}
Nov 02 12:38:47.738782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:47.738706    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podStartSLOduration=2.738683897 podStartE2EDuration="2.738683897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:47.737961576 +0000 UTC m=+725.387015535" watchObservedRunningTime="2026-11-02 12:38:47.738683897 +0000 UTC m=+725.387737858"
Nov 02 12:38:48.068287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.068259    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:38:48.069268 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:38:48.069241    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e8696c_fe05_45a5_90d3_b4343d48bda2.slice/crio-489dc32cf367978e8ba8974ec17798e8ff848a2a2a0d5c986a03a28023c199c6 WatchSource:0}: Error finding container 489dc32cf367978e8ba8974ec17798e8ff848a2a2a0d5c986a03a28023c199c6: Status 404 returned error can't find the container with id 489dc32cf367978e8ba8974ec17798e8ff848a2a2a0d5c986a03a28023c199c6
Nov 02 12:38:48.072438 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.072402    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:38:48.072512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.072484    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:38:48.072550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.072527    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:38:48.302383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.302349    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:48.698654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.698613    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" event={"ID":"48e8696c-fe05-45a5-90d3-b4343d48bda2","Type":"ContainerStarted","Data":"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"}
Nov 02 12:38:48.698853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.698675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" event={"ID":"48e8696c-fe05-45a5-90d3-b4343d48bda2","Type":"ContainerStarted","Data":"489dc32cf367978e8ba8974ec17798e8ff848a2a2a0d5c986a03a28023c199c6"}
Nov 02 12:38:48.753051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:48.745931    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podStartSLOduration=2.745911945 podStartE2EDuration="2.745911945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:48.745334366 +0000 UTC m=+726.394388346" watchObservedRunningTime="2026-11-02 12:38:48.745911945 +0000 UTC m=+726.394965904"
Nov 02 12:38:49.480025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:49.479993    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:49.481639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:49.481610    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:49.481779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:49.481660    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:50.480685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:50.480646    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:50.481111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:50.480712    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:51.480812 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:51.480773    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:51.481269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:51.480871    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:52.480419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:52.480383    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:52.480578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:52.480437    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:53.480416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:53.480378    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:53.480937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:53.480436    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:54.480311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:54.480277    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:38:54.480523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:54.480333    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 12:38:55.484245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:55.484212    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:55.484644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:55.484472    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:55.485220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:55.485185    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:38:56.302217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:56.302181    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:56.315326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:56.315294    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:38:56.805931 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:38:56.805899    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:39:01.121467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.121432    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:01.125776 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.125751    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.159478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159444    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.159658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159575    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.159767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159749    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.159849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159810    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.159936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159920    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.160017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.159998    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzv7\" (UniqueName: \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.177878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.177846    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"e2e-conflict-b-7590c91f-kserve-self-signed-certs\""
Nov 02 12:39:01.181954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.181926    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:01.261304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261260    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261317    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261377    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261419    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzv7\" (UniqueName: \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261659 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261490    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261659 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261556    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.261805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261776    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.262012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261976    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.262012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.261987    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.263864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.263814    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.264155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.264134    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.286675 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.286651    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzv7\" (UniqueName: \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\") pod \"e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") " pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.438950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.438851    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:01.752676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.752648    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:01.753958 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:39:01.753919    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a702282_20e5_40ef_97e8_0c394c6c5173.slice/crio-f4d8161a36cd7b1385963138a1a7e5ef36eb78ac3f777ebaa182875a35e68ade WatchSource:0}: Error finding container f4d8161a36cd7b1385963138a1a7e5ef36eb78ac3f777ebaa182875a35e68ade: Status 404 returned error can't find the container with id f4d8161a36cd7b1385963138a1a7e5ef36eb78ac3f777ebaa182875a35e68ade
Nov 02 12:39:01.828596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.828559    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" event={"ID":"0a702282-20e5-40ef-97e8-0c394c6c5173","Type":"ContainerStarted","Data":"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"}
Nov 02 12:39:01.828781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.828610    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" event={"ID":"0a702282-20e5-40ef-97e8-0c394c6c5173","Type":"ContainerStarted","Data":"f4d8161a36cd7b1385963138a1a7e5ef36eb78ac3f777ebaa182875a35e68ade"}
Nov 02 12:39:01.875227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:01.873779    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" podStartSLOduration=0.873759527 podStartE2EDuration="873.759527ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:01.872926614 +0000 UTC m=+739.521980573" watchObservedRunningTime="2026-11-02 12:39:01.873759527 +0000 UTC m=+739.522813486"
Nov 02 12:39:02.439502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:02.439464    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:04.082800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.082765    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:04.086996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.086974    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.106799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.106769    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:04.202683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.202641    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.202683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.202679    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49n8v\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.202922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.202704    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.202922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.202888    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.304475 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.304440    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.304475 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.304477    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49n8v\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.304744 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.304499    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.304744 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.304608    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.305353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.305301    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.305468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.305376    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.308117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.308065    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.329742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.329712    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49n8v\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\") pod \"payload-processing-external-model-7d66d6cd49-crnd6\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.401962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.401870    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.755528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.755494    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:04.759328 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:39:04.759300    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427546fc_8544_4527_b5c4_2d4cee110f7a.slice/crio-736b5de9882236305325bd54233914c4a117359456e25ad2f0cd5eca4e9b4f83 WatchSource:0}: Error finding container 736b5de9882236305325bd54233914c4a117359456e25ad2f0cd5eca4e9b4f83: Status 404 returned error can't find the container with id 736b5de9882236305325bd54233914c4a117359456e25ad2f0cd5eca4e9b4f83
Nov 02 12:39:04.855336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.855293    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" event={"ID":"427546fc-8544-4527-b5c4-2d4cee110f7a","Type":"ContainerStarted","Data":"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"}
Nov 02 12:39:04.855501 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.855369    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" event={"ID":"427546fc-8544-4527-b5c4-2d4cee110f7a","Type":"ContainerStarted","Data":"736b5de9882236305325bd54233914c4a117359456e25ad2f0cd5eca4e9b4f83"}
Nov 02 12:39:04.855564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.855519    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:04.904057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:04.904006    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" podStartSLOduration=0.903990151 podStartE2EDuration="903.990151ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:04.899303496 +0000 UTC m=+742.548357455" watchObservedRunningTime="2026-11-02 12:39:04.903990151 +0000 UTC m=+742.553044124"
Nov 02 12:39:09.729068 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:39:09.729023    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:39:11.439573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:11.439520    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:11.453024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:11.452999    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:11.917129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:11.917095    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:15.865726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:15.865694    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:15.957923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:15.957889    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:39:16.335849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:16.327615    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:20.958757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:20.958700    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" podUID="90983006-b8fe-4a63-b6ad-1808ba505438" containerName="payload-processing" containerID="cri-o://ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f" gracePeriod=25
Nov 02 12:39:21.238582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.238549    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:39:21.324738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.324700    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\") pod \"90983006-b8fe-4a63-b6ad-1808ba505438\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") "
Nov 02 12:39:21.324996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.324767    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\") pod \"90983006-b8fe-4a63-b6ad-1808ba505438\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") "
Nov 02 12:39:21.324996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.324940    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\") pod \"90983006-b8fe-4a63-b6ad-1808ba505438\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") "
Nov 02 12:39:21.325132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.325043    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\") pod \"90983006-b8fe-4a63-b6ad-1808ba505438\" (UID: \"90983006-b8fe-4a63-b6ad-1808ba505438\") "
Nov 02 12:39:21.325348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.325234    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume" pod "90983006-b8fe-4a63-b6ad-1808ba505438" (UID: "90983006-b8fe-4a63-b6ad-1808ba505438"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:21.325478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.325453    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume" pod "90983006-b8fe-4a63-b6ad-1808ba505438" (UID: "90983006-b8fe-4a63-b6ad-1808ba505438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:21.325600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.325577    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.325652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.325608    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90983006-b8fe-4a63-b6ad-1808ba505438-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.327979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.327942    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" podUID="427546fc-8544-4527-b5c4-2d4cee110f7a" containerName="payload-processing" containerID="cri-o://5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af" gracePeriod=25
Nov 02 12:39:21.328107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.328032    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials" pod "90983006-b8fe-4a63-b6ad-1808ba505438" (UID: "90983006-b8fe-4a63-b6ad-1808ba505438"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:21.328170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.328126    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9" pod "90983006-b8fe-4a63-b6ad-1808ba505438" (UID: "90983006-b8fe-4a63-b6ad-1808ba505438"). InnerVolumeSpecName "kube-api-access-m78q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:21.427352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.427313    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.427352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.427348    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78q9\" (UniqueName: \"kubernetes.io/projected/90983006-b8fe-4a63-b6ad-1808ba505438-kube-api-access-m78q9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.581372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.581345    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:21.731012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.730967    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\") pod \"427546fc-8544-4527-b5c4-2d4cee110f7a\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") "
Nov 02 12:39:21.731210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731080    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\") pod \"427546fc-8544-4527-b5c4-2d4cee110f7a\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") "
Nov 02 12:39:21.731210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731140    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\") pod \"427546fc-8544-4527-b5c4-2d4cee110f7a\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") "
Nov 02 12:39:21.731210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731196    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\") pod \"427546fc-8544-4527-b5c4-2d4cee110f7a\" (UID: \"427546fc-8544-4527-b5c4-2d4cee110f7a\") "
Nov 02 12:39:21.731475 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731449    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume" pod "427546fc-8544-4527-b5c4-2d4cee110f7a" (UID: "427546fc-8544-4527-b5c4-2d4cee110f7a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:21.731591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731565    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume" pod "427546fc-8544-4527-b5c4-2d4cee110f7a" (UID: "427546fc-8544-4527-b5c4-2d4cee110f7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:21.731809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731785    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.731809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.731811    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427546fc-8544-4527-b5c4-2d4cee110f7a-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.733161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.733138    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v" pod "427546fc-8544-4527-b5c4-2d4cee110f7a" (UID: "427546fc-8544-4527-b5c4-2d4cee110f7a"). InnerVolumeSpecName "kube-api-access-49n8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:21.733299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.733285    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials" pod "427546fc-8544-4527-b5c4-2d4cee110f7a" (UID: "427546fc-8544-4527-b5c4-2d4cee110f7a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:21.832543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.832443    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49n8v\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-kube-api-access-49n8v\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:21.832543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:21.832479    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/427546fc-8544-4527-b5c4-2d4cee110f7a-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:22.018401 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.018360    4928 generic.go:356] "Generic (PLEG): container finished" podID="427546fc-8544-4527-b5c4-2d4cee110f7a" containerID="5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af" exitCode=0
Nov 02 12:39:22.018902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.018414    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"
Nov 02 12:39:22.018902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.018444    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" event={"ID":"427546fc-8544-4527-b5c4-2d4cee110f7a","Type":"ContainerDied","Data":"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"}
Nov 02 12:39:22.018902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.018501    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6" event={"ID":"427546fc-8544-4527-b5c4-2d4cee110f7a","Type":"ContainerDied","Data":"736b5de9882236305325bd54233914c4a117359456e25ad2f0cd5eca4e9b4f83"}
Nov 02 12:39:22.018902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.018527    4928 scope.go:122] "RemoveContainer" containerID="5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"
Nov 02 12:39:22.020154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.020134    4928 generic.go:356] "Generic (PLEG): container finished" podID="90983006-b8fe-4a63-b6ad-1808ba505438" containerID="ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f" exitCode=0
Nov 02 12:39:22.020266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.020218    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" event={"ID":"90983006-b8fe-4a63-b6ad-1808ba505438","Type":"ContainerDied","Data":"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"}
Nov 02 12:39:22.020266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.020235    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt"
Nov 02 12:39:22.020266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.020245    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84545c695-chvlt" event={"ID":"90983006-b8fe-4a63-b6ad-1808ba505438","Type":"ContainerDied","Data":"87461891444df9bf275e419b53f5df3ffb5453725b8ad3aeb26c2ad9ccd351d9"}
Nov 02 12:39:22.032496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.029972    4928 scope.go:122] "RemoveContainer" containerID="5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"
Nov 02 12:39:22.032496 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:39:22.032085    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af\": container with ID starting with 5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af not found: ID does not exist" containerID="5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"
Nov 02 12:39:22.032496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.032123    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af"} err="failed to get container status \"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af\": rpc error: code = NotFound desc = could not find container \"5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af\": container with ID starting with 5c83b5a69d2c8427371bd5a642beca3948a8822ad0432579336a286da853e9af not found: ID does not exist"
Nov 02 12:39:22.032496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.032149    4928 scope.go:122] "RemoveContainer" containerID="ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"
Nov 02 12:39:22.049031 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.049014    4928 scope.go:122] "RemoveContainer" containerID="ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"
Nov 02 12:39:22.049339 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:39:22.049314    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f\": container with ID starting with ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f not found: ID does not exist" containerID="ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"
Nov 02 12:39:22.049433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.049344    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f"} err="failed to get container status \"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f\": rpc error: code = NotFound desc = could not find container \"ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f\": container with ID starting with ac407f2d57b94886da9e71d43e13c421116f800969ecc4c1e5dea7c45278646f not found: ID does not exist"
Nov 02 12:39:22.128178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.128092    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:39:22.141382 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.141346    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84545c695-chvlt"]
Nov 02 12:39:22.179380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.179336    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:22.185546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.185517    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-crnd6"]
Nov 02 12:39:22.497167 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.497132    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427546fc-8544-4527-b5c4-2d4cee110f7a" path="/var/lib/kubelet/pods/427546fc-8544-4527-b5c4-2d4cee110f7a/volumes"
Nov 02 12:39:22.497734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:22.497716    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90983006-b8fe-4a63-b6ad-1808ba505438" path="/var/lib/kubelet/pods/90983006-b8fe-4a63-b6ad-1808ba505438/volumes"
Nov 02 12:39:28.644558 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.644191    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 12:39:28.645175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645010    4928 cpu_manager.go:414] "containerMap: removing container" podUID="427546fc-8544-4527-b5c4-2d4cee110f7a" containerName="payload-processing"
Nov 02 12:39:28.645175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645050    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427546fc-8544-4527-b5c4-2d4cee110f7a" containerName="payload-processing"
Nov 02 12:39:28.645175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645088    4928 cpu_manager.go:414] "containerMap: removing container" podUID="90983006-b8fe-4a63-b6ad-1808ba505438" containerName="payload-processing"
Nov 02 12:39:28.645175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645096    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90983006-b8fe-4a63-b6ad-1808ba505438" containerName="payload-processing"
Nov 02 12:39:28.645379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645247    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="427546fc-8544-4527-b5c4-2d4cee110f7a" containerName="payload-processing"
Nov 02 12:39:28.645379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.645264    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="90983006-b8fe-4a63-b6ad-1808ba505438" containerName="payload-processing"
Nov 02 12:39:28.649222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.649197    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.671016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.670984    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 12:39:28.677201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.677176    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"kube-root-ca.crt\""
Nov 02 12:39:28.677336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.677314    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"default-dockercfg-h844f\""
Nov 02 12:39:28.677402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.677380    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"openshift-service-ca.crt\""
Nov 02 12:39:28.680537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.680516    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"distinct-2-w4-843978-kserve-self-signed-certs\""
Nov 02 12:39:28.745640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.745595    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxct\" (UniqueName: \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.745818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.745659    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.745818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.745727    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.745818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.745779    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.746014 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.745989    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.746075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.746048    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.846904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.846860    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxct\" (UniqueName: \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.847109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.846928    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.847109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.846954    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.847109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.847001    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.847281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.847129    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.847281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.847167    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.849399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.848557    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.849399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.848884    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.849399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.849121    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.865196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.865160    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.876114 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.876084    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.877966 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.877941    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxct\" (UniqueName: \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\") pod \"distinct-2-w4-843978-kserve-fc6455c9c-r86vn\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") " pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:28.969851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:28.969740    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:29.276656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:29.276629    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 12:39:29.278948 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:39:29.278918    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a1e00ed_5726_4d6e_807e_8ac7cec0fc06.slice/crio-7b234355ce650634aa61b821643c2e7e8655df953dd47c3cee4578682f685dd1 WatchSource:0}: Error finding container 7b234355ce650634aa61b821643c2e7e8655df953dd47c3cee4578682f685dd1: Status 404 returned error can't find the container with id 7b234355ce650634aa61b821643c2e7e8655df953dd47c3cee4578682f685dd1
Nov 02 12:39:30.114072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:30.114035    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" event={"ID":"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06","Type":"ContainerStarted","Data":"33957e8a6312ccc89a4b8f8e13f51d67905da99dea7c7f6c5fc720bd494ba128"}
Nov 02 12:39:30.114072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:30.114077    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" event={"ID":"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06","Type":"ContainerStarted","Data":"7b234355ce650634aa61b821643c2e7e8655df953dd47c3cee4578682f685dd1"}
Nov 02 12:39:30.163295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:30.163222    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podStartSLOduration=2.163200816 podStartE2EDuration="2.163200816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:30.157360925 +0000 UTC m=+767.806414884" watchObservedRunningTime="2026-11-02 12:39:30.163200816 +0000 UTC m=+767.812254775"
Nov 02 12:39:30.970130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:30.970089    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:38.970530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:38.970493    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:38.983271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:38.983239    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:39.201328 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:39.201295    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:39:39.785210 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:39:39.784581    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:39:40.455885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:40.453768    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:42.778200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.778168    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 12:39:42.783779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.783753    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.814751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.814710    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 12:39:42.975131 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975042    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975182    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975304    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975394    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975467    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975507    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975548    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975633 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975618    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmkd\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:42.975668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:42.975644    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.078849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.077373    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.078849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.077468    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.078923    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079045    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079152 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079087    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079152 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079126    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079175    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmkd\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079209    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079303    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079415 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079402    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079428    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079437    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.079584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.079510    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.080047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.080007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.082180 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.082154    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.083340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.083311    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.102530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.102500    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.103981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.103954    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmkd\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.402364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.402310    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:43.731845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.731801    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 12:39:43.733555 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:39:43.733526    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcd3a0c_108b_4136_bd14_4bd73e25b4d8.slice/crio-f01b62a28aa06bb5f423a2c65999ee5b0597afdca85374335d868d6e544c3d2d WatchSource:0}: Error finding container f01b62a28aa06bb5f423a2c65999ee5b0597afdca85374335d868d6e544c3d2d: Status 404 returned error can't find the container with id f01b62a28aa06bb5f423a2c65999ee5b0597afdca85374335d868d6e544c3d2d
Nov 02 12:39:43.736224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.736188    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:39:43.736312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.736261    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:39:43.736312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:43.736290    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:39:44.225550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.225506    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" event={"ID":"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8","Type":"ContainerStarted","Data":"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"}
Nov 02 12:39:44.226027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.225580    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" event={"ID":"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8","Type":"ContainerStarted","Data":"f01b62a28aa06bb5f423a2c65999ee5b0597afdca85374335d868d6e544c3d2d"}
Nov 02 12:39:44.306305 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.306252    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podStartSLOduration=2.306238928 podStartE2EDuration="2.306238928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:44.304677051 +0000 UTC m=+781.953731008" watchObservedRunningTime="2026-11-02 12:39:44.306238928 +0000 UTC m=+781.955292924"
Nov 02 12:39:44.403378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.403340    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:44.405244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.405217    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:44.405371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:44.405262    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:45.403132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:45.403095    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:45.403538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:45.403184    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:46.403008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:46.402975    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:46.403193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:46.403036    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:47.403302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:47.403265    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:47.403705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:47.403319    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:47.565110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:47.565068    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:39:48.403721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:48.403679    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:48.404145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:48.403746    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:49.403470 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:49.403429    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:49.403746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:49.403711    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:50.403279 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:50.403242    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:39:50.403463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:50.403310    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 12:39:52.445071 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:52.445018    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:39:52.445739 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:52.445699    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:39:52.511942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:52.511910    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:52.512340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:52.512307    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:52.513413 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:52.513391    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 12:39:55.479898 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.479848    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" podUID="0a702282-20e5-40ef-97e8-0c394c6c5173" containerName="main" containerID="cri-o://2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895" gracePeriod=45
Nov 02 12:39:55.754757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.754727    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:55.846151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.846119    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.846751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.846711    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\" (UniqueName: \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.846912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.846858    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.846994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.846911    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.847052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.847029    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.847110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.847061    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\") pod \"0a702282-20e5-40ef-97e8-0c394c6c5173\" (UID: \"0a702282-20e5-40ef-97e8-0c394c6c5173\") "
Nov 02 12:39:55.848111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.846593    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:39:55.848111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.847978    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:39:55.848111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.848053    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:39:55.850530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.850496    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:55.850638 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.850583    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:39:55.850714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.850692    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7" pod "0a702282-20e5-40ef-97e8-0c394c6c5173" (UID: "0a702282-20e5-40ef-97e8-0c394c6c5173"). InnerVolumeSpecName "kube-api-access-kfzv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:55.948479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948435    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:55.948479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948467    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:55.948479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948478    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfzv7\" (UniqueName: \"kubernetes.io/projected/0a702282-20e5-40ef-97e8-0c394c6c5173-kube-api-access-kfzv7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:55.948479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948488    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a702282-20e5-40ef-97e8-0c394c6c5173-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:55.948862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948496    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:55.948862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:55.948504    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a702282-20e5-40ef-97e8-0c394c6c5173-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:39:56.344141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.344101    4928 generic.go:356] "Generic (PLEG): container finished" podID="0a702282-20e5-40ef-97e8-0c394c6c5173" containerID="2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895" exitCode=0
Nov 02 12:39:56.344326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.344144    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" event={"ID":"0a702282-20e5-40ef-97e8-0c394c6c5173","Type":"ContainerDied","Data":"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"}
Nov 02 12:39:56.344326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.344201    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn" event={"ID":"0a702282-20e5-40ef-97e8-0c394c6c5173","Type":"ContainerDied","Data":"f4d8161a36cd7b1385963138a1a7e5ef36eb78ac3f777ebaa182875a35e68ade"}
Nov 02 12:39:56.344326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.344161    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"
Nov 02 12:39:56.344326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.344224    4928 scope.go:122] "RemoveContainer" containerID="2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"
Nov 02 12:39:56.355313 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.355291    4928 scope.go:122] "RemoveContainer" containerID="2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"
Nov 02 12:39:56.355755 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:39:56.355650    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895\": container with ID starting with 2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895 not found: ID does not exist" containerID="2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"
Nov 02 12:39:56.355755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.355680    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895"} err="failed to get container status \"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895\": rpc error: code = NotFound desc = could not find container \"2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895\": container with ID starting with 2a3f9ec585e4f724a241254766056851b0724d57eae3a2f0d5b73fd788e4b895 not found: ID does not exist"
Nov 02 12:39:56.403729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.403702    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:56.416937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.416912    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-b-7590c91f-kserve-674cd8686c-sq2jn"]
Nov 02 12:39:56.497025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:56.496994    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a702282-20e5-40ef-97e8-0c394c6c5173" path="/var/lib/kubelet/pods/0a702282-20e5-40ef-97e8-0c394c6c5173/volumes"
Nov 02 12:39:57.181010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.180981    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"]
Nov 02 12:39:57.181545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.181528    4928 cpu_manager.go:414] "containerMap: removing container" podUID="0a702282-20e5-40ef-97e8-0c394c6c5173" containerName="main"
Nov 02 12:39:57.181545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.181545    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a702282-20e5-40ef-97e8-0c394c6c5173" containerName="main"
Nov 02 12:39:57.181697 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.181682    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a702282-20e5-40ef-97e8-0c394c6c5173" containerName="main"
Nov 02 12:39:57.186584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.186558    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.202651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.202629    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:39:57.205467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.205443    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"]
Nov 02 12:39:57.215158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.215130    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:57.265765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.265736    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xvn\" (UniqueName: \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.265964 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.265893    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.266031 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.265964    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.367391 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.367354    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.367596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.367490    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.367596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.367551    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xvn\" (UniqueName: \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.368174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.368151    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.369684 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.369664    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.391896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.391870    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xvn\" (UniqueName: \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\") pod \"maas-api-revoke-sub-993035a6-vfk4g\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.502042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.502007    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:39:57.809166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:57.809135    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"]
Nov 02 12:39:57.811260 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:39:57.811208    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79616ada_c240_438f_9a2d_190563551866.slice/crio-11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a WatchSource:0}: Error finding container 11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a: Status 404 returned error can't find the container with id 11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a
Nov 02 12:39:58.362285 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:58.362243    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g" event={"ID":"79616ada-c240-438f-9a2d-190563551866","Type":"ContainerStarted","Data":"8fd2bea99df201980c32dbf843f35de6183aa498eee900e5a61beba22063e991"}
Nov 02 12:39:58.362467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:58.362310    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g" event={"ID":"79616ada-c240-438f-9a2d-190563551866","Type":"ContainerStarted","Data":"11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a"}
Nov 02 12:39:58.408621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:58.408504    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g" podStartSLOduration=1.408482983 podStartE2EDuration="1.408482983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:58.404146356 +0000 UTC m=+796.053200314" watchObservedRunningTime="2026-11-02 12:39:58.408482983 +0000 UTC m=+796.057536943"
Nov 02 12:39:59.371841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:59.371792    4928 generic.go:356] "Generic (PLEG): container finished" podID="79616ada-c240-438f-9a2d-190563551866" containerID="8fd2bea99df201980c32dbf843f35de6183aa498eee900e5a61beba22063e991" exitCode=0
Nov 02 12:39:59.372329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:39:59.371937    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g" event={"ID":"79616ada-c240-438f-9a2d-190563551866","Type":"ContainerDied","Data":"8fd2bea99df201980c32dbf843f35de6183aa498eee900e5a61beba22063e991"}
Nov 02 12:40:00.531883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.531850    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:40:00.606410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.606373    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\") pod \"79616ada-c240-438f-9a2d-190563551866\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") "
Nov 02 12:40:00.606655 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.606460    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\") pod \"79616ada-c240-438f-9a2d-190563551866\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") "
Nov 02 12:40:00.606655 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.606526    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\" (UniqueName: \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\") pod \"79616ada-c240-438f-9a2d-190563551866\" (UID: \"79616ada-c240-438f-9a2d-190563551866\") "
Nov 02 12:40:00.608040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.607998    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca" pod "79616ada-c240-438f-9a2d-190563551866" (UID: "79616ada-c240-438f-9a2d-190563551866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:00.620843 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.615963    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca" pod "79616ada-c240-438f-9a2d-190563551866" (UID: "79616ada-c240-438f-9a2d-190563551866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:00.632056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.631752    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn" pod "79616ada-c240-438f-9a2d-190563551866" (UID: "79616ada-c240-438f-9a2d-190563551866"). InnerVolumeSpecName "kube-api-access-d7xvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:00.708808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.708775    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79616ada-c240-438f-9a2d-190563551866-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:00.708808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.708806    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79616ada-c240-438f-9a2d-190563551866-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:00.708808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:00.708816    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7xvn\" (UniqueName: \"kubernetes.io/projected/79616ada-c240-438f-9a2d-190563551866-kube-api-access-d7xvn\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:01.388805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.388774    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"
Nov 02 12:40:01.389022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.388760    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g" event={"ID":"79616ada-c240-438f-9a2d-190563551866","Type":"ContainerDied","Data":"11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a"}
Nov 02 12:40:01.389022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.388939    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11099b11690b402d999321de122554e13449ba8682c044a57454ed97afa5d96a"
Nov 02 12:40:01.762195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.762159    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:40:01.762886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.762870    4928 cpu_manager.go:414] "containerMap: removing container" podUID="79616ada-c240-438f-9a2d-190563551866" containerName="revoke-keys"
Nov 02 12:40:01.762945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.762886    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79616ada-c240-438f-9a2d-190563551866" containerName="revoke-keys"
Nov 02 12:40:01.763025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.763012    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="79616ada-c240-438f-9a2d-190563551866" containerName="revoke-keys"
Nov 02 12:40:01.766936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.766916    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.794521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.794489    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:40:01.802897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.802868    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:40:01.808186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.808163    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.822501 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822466    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822522    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjcq\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822588    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822679    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822716    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822763    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822806    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822872    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.826244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.822898    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.829725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.829699    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:40:01.924487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924451    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.924689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924512    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.924689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924544    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.924689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924576    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.924947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924689    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.924947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924740    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.924947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924790    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.924947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924853    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.924947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924900    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924946    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924947    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.924996    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925037    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjcq\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925246    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925252    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.925389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925325    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925366    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.925693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925425    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925451    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8bb\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.925693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925566    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.925693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925623    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.925693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.925660    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:01.926125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.926102    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.927596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.927574    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.927695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.927676    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.949695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.949664    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:01.949817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:01.949738    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjcq\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:02.027555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027426    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027523    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027569    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027603    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027681    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027723    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.027816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027775    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8bb\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.028117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027819    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.028117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.027871    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.028280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.028259    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.028340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.028316    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.028533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.028506    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.029326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.029297    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.029592 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.029560    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.046875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.031875    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.046875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.033311    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.083887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.076926    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8bb\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.083887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.080439    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.083887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.080672    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:02.133164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.133115    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:02.444897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.444072    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:40:02.454169 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:40:02.451558    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816eae63_ce87_48e5_84d7_bfeec7c63a6c.slice/crio-008ac08befeedf4aa7586dd03291bce64e7943c3a9b82d1a1aa6d197ad5e43d4 WatchSource:0}: Error finding container 008ac08befeedf4aa7586dd03291bce64e7943c3a9b82d1a1aa6d197ad5e43d4: Status 404 returned error can't find the container with id 008ac08befeedf4aa7586dd03291bce64e7943c3a9b82d1a1aa6d197ad5e43d4
Nov 02 12:40:02.458356 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.458315    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.458491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.458430    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.458567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.458498    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.524852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.523601    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:40:02.539403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.539355    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.539545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.539461    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.539545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.539510    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:02.592402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:02.592247    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerName="main" containerID="cri-o://64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc" gracePeriod=45
Nov 02 12:40:03.573130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.573095    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" event={"ID":"c5b40528-7325-49d7-9b2d-edf9c0997dd6","Type":"ContainerStarted","Data":"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"}
Nov 02 12:40:03.573719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.573670    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" event={"ID":"c5b40528-7325-49d7-9b2d-edf9c0997dd6","Type":"ContainerStarted","Data":"ecd85c2aeee344cfa7d79c2286bdfc981ec6e3501c1fe66265c3db21edee69ee"}
Nov 02 12:40:03.577962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.574213    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:40:03.598884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.597849    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" event={"ID":"816eae63-ce87-48e5-84d7-bfeec7c63a6c","Type":"ContainerStarted","Data":"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"}
Nov 02 12:40:03.598884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.597914    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" event={"ID":"816eae63-ce87-48e5-84d7-bfeec7c63a6c","Type":"ContainerStarted","Data":"008ac08befeedf4aa7586dd03291bce64e7943c3a9b82d1a1aa6d197ad5e43d4"}
Nov 02 12:40:03.623926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.613604    4928 generic.go:356] "Generic (PLEG): container finished" podID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerID="64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc" exitCode=0
Nov 02 12:40:03.623926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.613786    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" event={"ID":"7a38229e-4c15-4b5c-a53c-a60d344f0ca4","Type":"ContainerDied","Data":"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"}
Nov 02 12:40:03.623926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.613843    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" event={"ID":"7a38229e-4c15-4b5c-a53c-a60d344f0ca4","Type":"ContainerDied","Data":"a1a452d3678aa368c873fc013970ea78cc2769d64be53b6944f8fe8ec06d20fc"}
Nov 02 12:40:03.623926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.613874    4928 scope.go:122] "RemoveContainer" containerID="64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"
Nov 02 12:40:03.623926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.614145    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"
Nov 02 12:40:03.650891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.650815    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podStartSLOduration=2.650794887 podStartE2EDuration="2.650794887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:03.640142009 +0000 UTC m=+801.289195980" watchObservedRunningTime="2026-11-02 12:40:03.650794887 +0000 UTC m=+801.299848845"
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673053    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\" (UniqueName: \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673132    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673164    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673211    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673338    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.676894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.673450    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\") pod \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\" (UID: \"7a38229e-4c15-4b5c-a53c-a60d344f0ca4\") "
Nov 02 12:40:03.699913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.699334    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "kube-api-access-lpssw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:03.719121 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.716201    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:40:03.724178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.724123    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:03.740013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.739937    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.776429    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podStartSLOduration=2.7764037139999997 podStartE2EDuration="2.776403714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:03.751416461 +0000 UTC m=+801.400470420" watchObservedRunningTime="2026-11-02 12:40:03.776403714 +0000 UTC m=+801.425457674"
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.790295    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.790331    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpssw\" (UniqueName: \"kubernetes.io/projected/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-kube-api-access-lpssw\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.790350    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.790363    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.796984    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:40:03.826865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.814732    4928 scope.go:122] "RemoveContainer" containerID="64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"
Nov 02 12:40:03.829529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.829485    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm" pod "7a38229e-4c15-4b5c-a53c-a60d344f0ca4" (UID: "7a38229e-4c15-4b5c-a53c-a60d344f0ca4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:40:03.850227 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:40:03.846060    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc\": container with ID starting with 64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc not found: ID does not exist" containerID="64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"
Nov 02 12:40:03.850227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.846113    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc"} err="failed to get container status \"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc\": rpc error: code = NotFound desc = could not find container \"64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc\": container with ID starting with 64bb930f851274baf18e61e292424680a78fb57740c0aeeaa3d7f75e4418befc not found: ID does not exist"
Nov 02 12:40:03.908911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.891279    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:03.908911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:03.906894    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38229e-4c15-4b5c-a53c-a60d344f0ca4-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.123206    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.135506    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.135574    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.138382    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.138480    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:04.138600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.138515    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:04.161458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.160690    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:40:04.184850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.181041    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4"]
Nov 02 12:40:04.246009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.245949    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:04.484028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.483982    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/e2e-conflict-a-7590c91f-kserve-66d57b9d9-589m4" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:04.694863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.693811    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" path="/var/lib/kubelet/pods/7a38229e-4c15-4b5c-a53c-a60d344f0ca4/volumes"
Nov 02 12:40:04.694863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.694326    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:40:04.695435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.695153    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerName="main"
Nov 02 12:40:04.695435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.695175    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerName="main"
Nov 02 12:40:04.695435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.695387    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a38229e-4c15-4b5c-a53c-a60d344f0ca4" containerName="main"
Nov 02 12:40:04.717877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.717124    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.719212 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.719178    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:40:04.749176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.735465    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-61e341-openshift-default-dockercfg-jcvsw\""
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.801522    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805082    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805151    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805263    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805292    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtnx\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805437    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805492    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805522    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.807564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.805552    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907203 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907162    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907236    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907271    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907356    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907382    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtnx\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907483    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907529    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907558    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.907658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.907588    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.908044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.908016    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.908215 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.908187    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.908390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.908225    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.908526 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.908235    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.908630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.908440    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.910154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.910128    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.910263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.910167    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.959852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.958848    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:04.969846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:04.961632    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtnx\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:05.088920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.086256    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:05.088920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.086319    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:05.111117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.111085    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:05.144850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.141324    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:05.144850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.141385    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:05.215392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.211513    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 12:40:05.217713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.217680    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.265668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.264146    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"distinct-w3-068756-kserve-self-signed-certs\""
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.308703    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313337    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313399    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313508    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74w6\" (UniqueName: \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313563    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313621    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.315962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.313665    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.414799    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.414893    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.415023    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74w6\" (UniqueName: \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.415077    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.415114    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.415157    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.415654    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.417704    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.418083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.417988    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.447845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.435343    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.604049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.604010    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.604548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.604524    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74w6\" (UniqueName: \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\") pod \"distinct-w3-068756-kserve-6598b674d6-cfghb\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") " pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.621604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.621323    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:05.653213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.625407    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:40:05.695587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.695550    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:05.695928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.695620    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:05.695928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.695653    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:05.793887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:05.789292    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" event={"ID":"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0","Type":"ContainerStarted","Data":"2fc88f3346cb2a2893469bbb5974250db056c36ec3df774fccae25af69ab22aa"}
Nov 02 12:40:06.095543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.095489    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:06.095884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.095818    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:06.102137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.102097    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 12:40:06.110005 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:40:06.109952    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ea8f8b_5b83_4c32_8d1c_2c04b4b85fa5.slice/crio-f6432d00165f2b28e2e4f58490a6a8299fa2ba357dbd2729e5deeb4f08b0f8d6 WatchSource:0}: Error finding container f6432d00165f2b28e2e4f58490a6a8299fa2ba357dbd2729e5deeb4f08b0f8d6: Status 404 returned error can't find the container with id f6432d00165f2b28e2e4f58490a6a8299fa2ba357dbd2729e5deeb4f08b0f8d6
Nov 02 12:40:06.170851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.149667    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:06.170851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.149727    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:06.802130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.801863    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" event={"ID":"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5","Type":"ContainerStarted","Data":"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"}
Nov 02 12:40:06.802130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.801937    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" event={"ID":"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5","Type":"ContainerStarted","Data":"f6432d00165f2b28e2e4f58490a6a8299fa2ba357dbd2729e5deeb4f08b0f8d6"}
Nov 02 12:40:06.804146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.804116    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" event={"ID":"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0","Type":"ContainerStarted","Data":"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"}
Nov 02 12:40:06.909169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.909093    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podStartSLOduration=2.909075442 podStartE2EDuration="2.909075442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:06.908329478 +0000 UTC m=+804.557383437" watchObservedRunningTime="2026-11-02 12:40:06.909075442 +0000 UTC m=+804.558129404"
Nov 02 12:40:06.917130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:06.912954    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podStartSLOduration=1.91293683 podStartE2EDuration="1.91293683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:06.861145128 +0000 UTC m=+804.510199087" watchObservedRunningTime="2026-11-02 12:40:06.91293683 +0000 UTC m=+804.561990790"
Nov 02 12:40:07.092979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.092379    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:07.092979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.092458    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:07.120161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.120113    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:07.121959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.121924    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:07.122122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.121991    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:07.157847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.157389    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:07.157847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.157456    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:07.622174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:07.622129    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:08.081849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.081461    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:08.081849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.081523    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:08.112632 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.112486    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:08.112632 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.112570    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:08.171850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.170977    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:08.171850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:08.171041    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:09.082851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.082547    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:09.082851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.082613    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:09.111995 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.111963    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:09.112208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.112178    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:09.154846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.154210    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:09.154846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.154268    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:09.784643 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:40:09.784429    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:40:09.810864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.810806    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"]
Nov 02 12:40:09.815015 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.814984    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:09.870130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.869483    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:40:09.870130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.869982    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:09.896512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:09.896470    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"]
Nov 02 12:40:10.014873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.014811    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnsq\" (UniqueName: \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.014873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.014875    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.015161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.014977    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.081423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.081335    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:40:10.081423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.081395    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 12:40:10.112041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.112006    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:10.112440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.112057    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:10.116724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.116691    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnsq\" (UniqueName: \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.116864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.116753    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.116864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.116797    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.117797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.117769    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.119269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.119245    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.157274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.156015    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:40:10.157274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.156074    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 12:40:10.164744 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.164694    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnsq\" (UniqueName: \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\") pod \"maas-api-revoke-sub-faea81e9-9dhvv\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.441902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.441022    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:10.793894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.793865    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"]
Nov 02 12:40:10.953339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:10.953302    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv" event={"ID":"4e229eda-d1ff-4fec-8c01-b3cac6221902","Type":"ContainerStarted","Data":"d3c3e03ff5f012a352b6e079e43999d507da69193ddc4e8232075ef599ef7f66"}
Nov 02 12:40:11.113702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.113607    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:11.114334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.114297    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:11.612850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.611882    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:11.613911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.613788    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:11.614461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.614440    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:40:11.800168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.797461    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:11.800168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.798593    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:11.802250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.801909    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:40:11.976204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:11.976167    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv" event={"ID":"4e229eda-d1ff-4fec-8c01-b3cac6221902","Type":"ContainerStarted","Data":"dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169"}
Nov 02 12:40:12.022174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.022109    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv" podStartSLOduration=3.022090521 podStartE2EDuration="3.022090521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:12.0183341 +0000 UTC m=+809.667388058" watchObservedRunningTime="2026-11-02 12:40:12.022090521 +0000 UTC m=+809.671144546"
Nov 02 12:40:12.112254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.112151    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:12.112254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.112216    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:12.269521 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:40:12.269483    4928 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e229eda_d1ff_4fec_8c01_b3cac6221902.slice/crio-dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e229eda_d1ff_4fec_8c01_b3cac6221902.slice/crio-conmon-dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:40:12.380986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.380880    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:40:12.384987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.384970    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.408759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.408727    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457094    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457180    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457282    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457358    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxh9l\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457425    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457459    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457509    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457565    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.457849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.457693    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558576    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558646    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558732    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558791    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558841    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.558940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558916    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxh9l\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.558983    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559015    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559065    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559073    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559162    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559438    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559670    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.559700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.559694    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.561542 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.561519    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.561724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.561707    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.583434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.583412    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.583771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.583744    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxh9l\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.702505 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.702408    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:12.987221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.987179    4928 generic.go:356] "Generic (PLEG): container finished" podID="4e229eda-d1ff-4fec-8c01-b3cac6221902" containerID="dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169" exitCode=0
Nov 02 12:40:12.987426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:12.987270    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv" event={"ID":"4e229eda-d1ff-4fec-8c01-b3cac6221902","Type":"ContainerDied","Data":"dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169"}
Nov 02 12:40:13.030279 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.030248    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:40:13.032918 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:40:13.032884    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4895448_e75c_4614_8bbd_39b2538bb7a9.slice/crio-24a40d496c60d3e75e37cd394a46ca1ad2febb9a0cbd814bea8342113c937edc WatchSource:0}: Error finding container 24a40d496c60d3e75e37cd394a46ca1ad2febb9a0cbd814bea8342113c937edc: Status 404 returned error can't find the container with id 24a40d496c60d3e75e37cd394a46ca1ad2febb9a0cbd814bea8342113c937edc
Nov 02 12:40:13.035778 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.035737    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:13.035892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.035855    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:13.035950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.035902    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:13.112888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.112741    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 12:40:13.112888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.112818    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 12:40:13.996698 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.996659    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" event={"ID":"e4895448-e75c-4614-8bbd-39b2538bb7a9","Type":"ContainerStarted","Data":"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"}
Nov 02 12:40:13.997220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:13.996727    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" event={"ID":"e4895448-e75c-4614-8bbd-39b2538bb7a9","Type":"ContainerStarted","Data":"24a40d496c60d3e75e37cd394a46ca1ad2febb9a0cbd814bea8342113c937edc"}
Nov 02 12:40:14.079907 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.079849    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podStartSLOduration=2.079820747 podStartE2EDuration="2.079820747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:14.062469027 +0000 UTC m=+811.711522984" watchObservedRunningTime="2026-11-02 12:40:14.079820747 +0000 UTC m=+811.728874705"
Nov 02 12:40:14.118385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.118353    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:14.119286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.119103    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:14.119853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.119819    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:40:14.218299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.218271    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:14.283303 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.283204    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\" (UniqueName: \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\") pod \"4e229eda-d1ff-4fec-8c01-b3cac6221902\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") "
Nov 02 12:40:14.283737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.283713    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\") pod \"4e229eda-d1ff-4fec-8c01-b3cac6221902\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") "
Nov 02 12:40:14.283905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.283889    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\") pod \"4e229eda-d1ff-4fec-8c01-b3cac6221902\" (UID: \"4e229eda-d1ff-4fec-8c01-b3cac6221902\") "
Nov 02 12:40:14.285044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.285018    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca" pod "4e229eda-d1ff-4fec-8c01-b3cac6221902" (UID: "4e229eda-d1ff-4fec-8c01-b3cac6221902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:14.287684 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.287655    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq" pod "4e229eda-d1ff-4fec-8c01-b3cac6221902" (UID: "4e229eda-d1ff-4fec-8c01-b3cac6221902"). InnerVolumeSpecName "kube-api-access-wqnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:14.288242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.288219    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca" pod "4e229eda-d1ff-4fec-8c01-b3cac6221902" (UID: "4e229eda-d1ff-4fec-8c01-b3cac6221902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:14.385455 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.385420    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:14.385455 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.385453    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e229eda-d1ff-4fec-8c01-b3cac6221902-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:14.385657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.385465    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqnsq\" (UniqueName: \"kubernetes.io/projected/4e229eda-d1ff-4fec-8c01-b3cac6221902-kube-api-access-wqnsq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:40:14.702976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.702866    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:14.704837 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.704804    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:14.704921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:14.704867    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:15.008886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.008856    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"
Nov 02 12:40:15.008886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.008855    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv" event={"ID":"4e229eda-d1ff-4fec-8c01-b3cac6221902","Type":"ContainerDied","Data":"d3c3e03ff5f012a352b6e079e43999d507da69193ddc4e8232075ef599ef7f66"}
Nov 02 12:40:15.009338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.008928    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c3e03ff5f012a352b6e079e43999d507da69193ddc4e8232075ef599ef7f66"
Nov 02 12:40:15.622322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.622285    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:15.670636 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.670606    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:15.703150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.703114    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:15.703322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:15.703178    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:16.065462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:16.065430    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:16.703679 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:16.703642    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:16.703875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:16.703700    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:17.703199 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:17.703163    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:17.703664 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:17.703227    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:18.703163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:18.703123    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:18.703320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:18.703195    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:19.704198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:19.704158    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:40:19.704890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:19.704854    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 12:40:21.701982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:21.701928    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:21.750006 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:21.749957    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:40:21.750269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:21.750236    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:21.791046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:21.790983    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:21.796442 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:21.796212    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.023024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.022969    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.190972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.190910    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.235130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.235089    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:40:22.320385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.320268    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:22.322171 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.322146    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:22.338169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.338124    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:40:22.704176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.704072    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.873164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.873114    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 12:40:22.939383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.900935    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.963461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:22.958244    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:23.043173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.043126    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:23.253923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.253871    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:23.289671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.289287    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:23.327436 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.327381    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:40:23.328057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.328035    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:40:23.329606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.329588    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:40:23.329780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.329766    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:23.329999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.329985    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:40:23.350439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.350418    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:40:23.379268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.379154    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:40:23.393966 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.393941    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:23.415764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.415737    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:40:23.416286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:23.416268    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:40:27.909312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.909277    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:40:27.909902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.909871    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4e229eda-d1ff-4fec-8c01-b3cac6221902" containerName="revoke-keys"
Nov 02 12:40:27.909902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.909905    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e229eda-d1ff-4fec-8c01-b3cac6221902" containerName="revoke-keys"
Nov 02 12:40:27.910023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.910013    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e229eda-d1ff-4fec-8c01-b3cac6221902" containerName="revoke-keys"
Nov 02 12:40:27.913214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.913191    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.946978    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947206 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947109    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqzm\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947206 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947153    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947272    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947420 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947401    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947502    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947593 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947573    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947610    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.947696 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.947641    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:27.948610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:27.948539    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:40:28.048968 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.048925    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049005    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049082    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049107    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049167    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049236    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqzm\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049266    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049485 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049342    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049524    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049756    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049812    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049862    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.049971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.049886    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.051689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.051667    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.051769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.051700    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.074468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.074439    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.074658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.074635    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqzm\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.227255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.227157    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:28.553928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.553898    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:40:28.555172 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:40:28.555141    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609e74f0_7cf8_4324_8c10_ba5f4e16ba80.slice/crio-e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280 WatchSource:0}: Error finding container e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280: Status 404 returned error can't find the container with id e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280
Nov 02 12:40:28.558446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.558402    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:28.558541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.558491    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:28.558541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:28.558533    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:40:29.408424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:29.408389    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" event={"ID":"609e74f0-7cf8-4324-8c10-ba5f4e16ba80","Type":"ContainerStarted","Data":"d9f35cd6c6c1a77848d238435735b05ba6d489b4f08a07a8fe3f78029b80ae5b"}
Nov 02 12:40:29.408867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:29.408464    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" event={"ID":"609e74f0-7cf8-4324-8c10-ba5f4e16ba80","Type":"ContainerStarted","Data":"e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280"}
Nov 02 12:40:29.472700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:29.472639    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podStartSLOduration=2.472621157 podStartE2EDuration="2.472621157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:29.471291882 +0000 UTC m=+827.120345839" watchObservedRunningTime="2026-11-02 12:40:29.472621157 +0000 UTC m=+827.121675115"
Nov 02 12:40:30.234850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:30.230846    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:30.234850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:30.230977    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:40:30.234850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:30.231021    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 12:40:31.233170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:31.233127    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:40:31.233757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:31.233722    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 12:40:32.232853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:32.230173    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:40:32.232853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:32.230241    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 12:40:33.238928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:33.229184    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:40:33.238928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:33.229248    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 12:40:34.244861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:34.229138    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:40:34.244861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:34.229202    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 12:40:35.518178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:35.518135    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:35.518713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:35.518681    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:35.519671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:35.519653    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:40:38.601904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.601863    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 12:40:38.625205 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.625174    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 12:40:38.625545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.625488    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.669701 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.669669    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"distinct-2-w3-068756-kserve-self-signed-certs\""
Nov 02 12:40:38.755849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.751819    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.786975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.786933    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.787264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.787240    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.806985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.806943    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.807425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.807398    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.833850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.810923    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkcz\" (UniqueName: \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.973851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.973735    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.974110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.974087    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.974390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.974369    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.974552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.974532    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkcz\" (UniqueName: \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.985879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.974726    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.986112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.986087    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.986329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.979990    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.986452 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.980221    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:38.986562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:38.979742    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:39.071850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:39.031530    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:39.072452 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:39.072417    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:39.160014 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:39.159976    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkcz\" (UniqueName: \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\") pod \"distinct-2-w3-068756-kserve-756647c76f-4fsmk\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") " pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:39.163846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:39.160443    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:39.964997 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:40:39.958737    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:40:39.973980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:39.973928    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:40.380521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:40.380490    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 12:40:40.381749 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:40:40.381720    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41d1b96_094b_4bc1_ad11_684ca5244607.slice/crio-a2abffbadd70c4167fcf2b136aa30cb6ba89759d6cc51e4de6838f8aece55c34 WatchSource:0}: Error finding container a2abffbadd70c4167fcf2b136aa30cb6ba89759d6cc51e4de6838f8aece55c34: Status 404 returned error can't find the container with id a2abffbadd70c4167fcf2b136aa30cb6ba89759d6cc51e4de6838f8aece55c34
Nov 02 12:40:40.750570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:40.750522    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" event={"ID":"f41d1b96-094b-4bc1-ad11-684ca5244607","Type":"ContainerStarted","Data":"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"}
Nov 02 12:40:40.750779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:40.750586    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" event={"ID":"f41d1b96-094b-4bc1-ad11-684ca5244607","Type":"ContainerStarted","Data":"a2abffbadd70c4167fcf2b136aa30cb6ba89759d6cc51e4de6838f8aece55c34"}
Nov 02 12:40:40.795983 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:40.795924    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podStartSLOduration=2.795909859 podStartE2EDuration="2.795909859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:40.794369161 +0000 UTC m=+838.443423118" watchObservedRunningTime="2026-11-02 12:40:40.795909859 +0000 UTC m=+838.444963816"
Nov 02 12:40:41.161376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:41.161283    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:49.161329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:49.161290    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:49.176518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:49.176492    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:49.830028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:49.829993    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:56.410250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:56.405964    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:56.425456 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:56.425415    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:56.637851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:56.632936    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:56.897851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:56.871912    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.316859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.314976    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.401849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.398930    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.409847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.408216    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.409847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.408524    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:40:57.435909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.427653    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 12:40:57.435909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.428198    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:40:57.452849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.451938    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.837179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.837122    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.846993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.846953    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:40:57.847309 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.847262    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.878479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.878438    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:40:57.905804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:57.905770    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:40:58.188010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.184984    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.258981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.258936    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 12:40:58.399853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.399765    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.429851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.426676    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:40:58.429851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.427878    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.429851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.428147    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.429851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.428328    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 12:40:58.482116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.461317    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.482116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.462688    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:58.850858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.849129    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.850858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.849974    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:40:58.914086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:58.914036    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.161506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.161418    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:40:59.300861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.282899    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.300861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.283334    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:40:59.408849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.402937    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 12:40:59.455584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.439114    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.455584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.439504    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.455584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.439770    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.455584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.440064    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:40:59.581850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.558736    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.658361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.657259    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:40:59.819855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.819806    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:40:59.823598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.823567    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:40:59.852980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.852939    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.880380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.880352    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:40:59.880947 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.880921    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:40:59.882749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.882721    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:40:59.895840 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:40:59.895681    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:41:03.405189 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.405154    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"]
Nov 02 12:41:03.409376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.409349    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.424429 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.424397    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcfk\" (UniqueName: \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.424620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.424606    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.424620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.424629    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.430388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.430358    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"]
Nov 02 12:41:03.438548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.438511    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:41:03.438675 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.438578    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:41:03.525805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.525771    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.525805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.525807    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.525805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.525890    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcfk\" (UniqueName: \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.526669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.526642    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.528638 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.528612    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.554331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.554299    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcfk\" (UniqueName: \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\") pod \"maas-api-revoke-sub-0a6efeb9-5xp28\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:03.723311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:03.723211    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:04.032799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.032766    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"]
Nov 02 12:41:04.034519 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:04.034482    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0377b40_6d16_4b21_a005_914f934788cb.slice/crio-4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b WatchSource:0}: Error finding container 4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b: Status 404 returned error can't find the container with id 4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b
Nov 02 12:41:04.661282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.661179    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28" event={"ID":"c0377b40-6d16-4b21-a005-914f934788cb","Type":"ContainerStarted","Data":"d41671a1ed070358106f68d631bf94841278b75f62cefe1564181668b351e2f7"}
Nov 02 12:41:04.661282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.661244    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28" event={"ID":"c0377b40-6d16-4b21-a005-914f934788cb","Type":"ContainerStarted","Data":"4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b"}
Nov 02 12:41:04.707241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.707183    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28" podStartSLOduration=1.707168336 podStartE2EDuration="1.707168336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:04.704247347 +0000 UTC m=+862.353301309" watchObservedRunningTime="2026-11-02 12:41:04.707168336 +0000 UTC m=+862.356222293"
Nov 02 12:41:04.735875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.735836    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 12:41:04.739704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.739676    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.763588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.763561    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 12:41:04.843917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.843877    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.843998    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844086 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844068    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g9j\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844095    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844199 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844166    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844221    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844271    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844352    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.844466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.844451    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.945902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.945791    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.945959    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946107    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946226    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946290    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946329    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g9j\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946358    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946434 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946406    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946448    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946573    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.946863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.946841    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.947154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.947129    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.947517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.947478    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.947642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.947600    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.948333 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.948311    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.948754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.948736    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.971601 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.971574    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:04.972051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:04.972030    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g9j\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:05.054674 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.054636    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:05.378803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.378769    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 12:41:05.379771 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:05.379743    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432729fb_e115_49e2_908a_db3f8309da85.slice/crio-d5098cbcd199053315a968dbddb14d41eb879c6078810431e0635b964bd3b66f WatchSource:0}: Error finding container d5098cbcd199053315a968dbddb14d41eb879c6078810431e0635b964bd3b66f: Status 404 returned error can't find the container with id d5098cbcd199053315a968dbddb14d41eb879c6078810431e0635b964bd3b66f
Nov 02 12:41:05.382775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.382738    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:05.382926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.382804    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:05.382926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.382849    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:05.670654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.670572    4928 generic.go:356] "Generic (PLEG): container finished" podID="c0377b40-6d16-4b21-a005-914f934788cb" containerID="d41671a1ed070358106f68d631bf94841278b75f62cefe1564181668b351e2f7" exitCode=0
Nov 02 12:41:05.671097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.670663    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28" event={"ID":"c0377b40-6d16-4b21-a005-914f934788cb","Type":"ContainerDied","Data":"d41671a1ed070358106f68d631bf94841278b75f62cefe1564181668b351e2f7"}
Nov 02 12:41:05.672679 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.672646    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" event={"ID":"432729fb-e115-49e2-908a-db3f8309da85","Type":"ContainerStarted","Data":"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"}
Nov 02 12:41:05.672800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.672697    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" event={"ID":"432729fb-e115-49e2-908a-db3f8309da85","Type":"ContainerStarted","Data":"d5098cbcd199053315a968dbddb14d41eb879c6078810431e0635b964bd3b66f"}
Nov 02 12:41:05.764142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:05.764088    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podStartSLOduration=1.764071435 podStartE2EDuration="1.764071435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:05.761286775 +0000 UTC m=+863.410340732" watchObservedRunningTime="2026-11-02 12:41:05.764071435 +0000 UTC m=+863.413125405"
Nov 02 12:41:06.055447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.055410    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:06.057410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.057379    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:06.057531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.057432    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:06.905437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.905407    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.972715    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\" (UniqueName: \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\") pod \"c0377b40-6d16-4b21-a005-914f934788cb\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") "
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.973153    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\") pod \"c0377b40-6d16-4b21-a005-914f934788cb\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") "
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.973555    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca" pod "c0377b40-6d16-4b21-a005-914f934788cb" (UID: "c0377b40-6d16-4b21-a005-914f934788cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.973707    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\") pod \"c0377b40-6d16-4b21-a005-914f934788cb\" (UID: \"c0377b40-6d16-4b21-a005-914f934788cb\") "
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.974595    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:41:06.979359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.975508    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk" pod "c0377b40-6d16-4b21-a005-914f934788cb" (UID: "c0377b40-6d16-4b21-a005-914f934788cb"). InnerVolumeSpecName "kube-api-access-ttcfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:41:06.979994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:06.979965    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca" pod "c0377b40-6d16-4b21-a005-914f934788cb" (UID: "c0377b40-6d16-4b21-a005-914f934788cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:41:07.055385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.055339    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:07.055592 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.055422    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:07.075532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.075491    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0377b40-6d16-4b21-a005-914f934788cb-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:41:07.075532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.075526    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttcfk\" (UniqueName: \"kubernetes.io/projected/c0377b40-6d16-4b21-a005-914f934788cb-kube-api-access-ttcfk\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:41:07.688811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.688775    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"
Nov 02 12:41:07.688811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.688794    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28" event={"ID":"c0377b40-6d16-4b21-a005-914f934788cb","Type":"ContainerDied","Data":"4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b"}
Nov 02 12:41:07.689115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:07.688857    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae09377abe35ffb3c3b7f509dca2b22ddee7e406ecb5411c3bcc6ee463ff92b"
Nov 02 12:41:08.059288 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:08.059186    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:08.059288 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:08.059248    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:09.058291 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:09.058251    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:09.058588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:09.058557    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:09.761216 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:41:09.761157    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:41:10.056535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:10.056433    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:10.056850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:10.056801    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:11.058262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.058219    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:11.058791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.058299    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:11.085745 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.085701    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 12:41:11.086450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.086428    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c0377b40-6d16-4b21-a005-914f934788cb" containerName="revoke-keys"
Nov 02 12:41:11.086450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.086450    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0377b40-6d16-4b21-a005-914f934788cb" containerName="revoke-keys"
Nov 02 12:41:11.086637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.086621    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0377b40-6d16-4b21-a005-914f934788cb" containerName="revoke-keys"
Nov 02 12:41:11.090389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.090363    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.110923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.110895    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 12:41:11.120087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.120053    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"openshift-service-ca.crt\""
Nov 02 12:41:11.120087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.120056    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"kube-root-ca.crt\""
Nov 02 12:41:11.120275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.120086    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"distinct-2-w2-95ee26-kserve-self-signed-certs\""
Nov 02 12:41:11.120275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.120180    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"default-dockercfg-gm7qj\""
Nov 02 12:41:11.163222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163182    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.163584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163554    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.163722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163653    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hkr\" (UniqueName: \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.163812 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163775    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.163901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163813    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.163963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.163905    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.268860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.268661    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.268860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.268743    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hkr\" (UniqueName: \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.268860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.268809    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.275855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.269244    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.275855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.269333    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.275855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.269468    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.276135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.275923    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.276844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.276257    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.276844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.276542    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.277484 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.277140    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.297495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.297459    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.333074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.324183    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hkr\" (UniqueName: \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\") pod \"distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.406852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.406664    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:11.723875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:11.723844    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 12:41:11.726965 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:11.726930    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4312071_44b5_4b2c_9e6c_5b9d8c677f3e.slice/crio-1e62a1201afcfda337ee95e1685a738a68ad2c85513b73230c69999c39c0d188 WatchSource:0}: Error finding container 1e62a1201afcfda337ee95e1685a738a68ad2c85513b73230c69999c39c0d188: Status 404 returned error can't find the container with id 1e62a1201afcfda337ee95e1685a738a68ad2c85513b73230c69999c39c0d188
Nov 02 12:41:12.055561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.055515    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:41:12.055749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.055569    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 12:41:12.375487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.375401    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:41:12.379999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.379969    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.406393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.406361    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:41:12.489345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489297    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx22j\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489410    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489453    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489651    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489697    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489803    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.489972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.489952    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.490034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.490006    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.490117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.490100    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591324    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591380    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591431    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591495    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx22j\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591534    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591565    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591711    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591742    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.591896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591804    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.592056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.591939    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.592295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.592250    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.592412 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.592389    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.593660 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.593582    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.593792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.593774    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.607231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.607198    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.610433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.610363    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.616388 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.616362    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.616973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.616952    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx22j\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.697503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.697415    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:12.733341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.733303    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" event={"ID":"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e","Type":"ContainerStarted","Data":"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"}
Nov 02 12:41:12.733581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.733355    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" event={"ID":"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e","Type":"ContainerStarted","Data":"1e62a1201afcfda337ee95e1685a738a68ad2c85513b73230c69999c39c0d188"}
Nov 02 12:41:12.798425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:12.796845    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podStartSLOduration=1.7968158440000002 podStartE2EDuration="1.796815844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:12.791723787 +0000 UTC m=+870.440777745" watchObservedRunningTime="2026-11-02 12:41:12.796815844 +0000 UTC m=+870.445869801"
Nov 02 12:41:13.074647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.064581    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:13.074647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.065671    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:13.074647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.067395    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 12:41:13.074647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.070115    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:41:13.083109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.083065    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:13.083243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.083136    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:13.083243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.083165    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:13.412914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.410459    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:13.850868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.848912    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" event={"ID":"ffcb3c32-3642-45c5-b99c-54497f8bcf40","Type":"ContainerStarted","Data":"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"}
Nov 02 12:41:13.850868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:13.848980    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" event={"ID":"ffcb3c32-3642-45c5-b99c-54497f8bcf40","Type":"ContainerStarted","Data":"086905c53771f10921b0ceb21847a89a1fb8130585dfadd87e1c9ee746e167e2"}
Nov 02 12:41:14.700913 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:14.700802    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:14.701896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:14.701519    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:14.701896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:14.701594    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:14.842864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:14.839766    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:14.988937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:14.988890    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.184847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.174961    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.313002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.312902    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.448886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.435912    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.460852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.457142    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.664025 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.663914    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:41:15.700293 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.700250    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:15.700579 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.700547    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:15.842948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.842903    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.843940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.843916    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:41:15.891798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.891732    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podStartSLOduration=3.891712985 podStartE2EDuration="3.891712985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:13.908315554 +0000 UTC m=+871.557369518" watchObservedRunningTime="2026-11-02 12:41:15.891712985 +0000 UTC m=+873.540766944"
Nov 02 12:41:15.987035 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.986983    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.999976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:15.999915    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.069850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.004289    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.086269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.048605    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:41:16.111851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.107919    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:16.157851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.132963    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.306992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.225037    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 12:41:16.306992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.225929    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:41:16.440875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.436701    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.467892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.447306    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:41:16.501614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.476114    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.501614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.476631    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:41:16.654932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.634152    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Nov 02 12:41:16.685979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.685931    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.760040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.760001    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:16.866857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.808237    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:16.866857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.793737    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:41:16.881989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.881930    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.991908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.988921    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.995347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:16.995313    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:41:17.046965 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.046907    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.087536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.087225    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.096018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.095979    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:41:17.270945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.233632    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.343709 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.343654    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.439118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.437946    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.492977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.492928    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.697560 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.696199    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:41:17.743684 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.743644    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:17.743973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.743941    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:17.883012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:17.882959    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.031946 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.031904    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.056379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.056337    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.127336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.127287    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.208321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.208274    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.276979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.276926    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.377089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.373931    4928 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:18.377089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.374001    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.404851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.395932    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:18.404851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.396001    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.448880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.447518    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.499667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.499626    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.708611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.708506    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:18.708884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.708853    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:18.723863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.711789    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.899854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:18.887372    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 12:41:19.043311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.043269    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 12:41:19.060997 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.060948    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.264880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.257903    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.304839 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.304719    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.461851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.447996    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 12:41:19.512424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.512362    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.563385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.563266    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.698502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.698465    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:19.698785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.698757    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:19.728740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.728701    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.888036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.887929    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.981859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:19.980926    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Nov 02 12:41:20.054849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.054354    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.070480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.070434    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.264910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.261159    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.271934 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.268610    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded"
Nov 02 12:41:20.327127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.327083    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.450950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.450901    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.520021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.519894    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.702849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.699165    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:20.702849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.699225    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:20.737963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.737909    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.891914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:20.888987    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.068885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.063870    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.143911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.133594    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.277851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.272303    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.344852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.325166    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:41:21.346327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.346283    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.418694 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.417777    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:21.452590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.451908    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.520271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.520236    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:41:21.521985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.521963    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:41:21.550876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.543904    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.621959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.621929    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:41:21.622960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.622930    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:41:21.641995 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.640410    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:41:21.644108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.644087    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:41:21.661446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.658986    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:41:21.661446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.659061    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:21.698276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.698185    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:21.698501 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.698475    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:21.912739 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:21.912708    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:41:22.697888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:22.697814    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:22.698352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:22.697912    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:23.698453 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:23.698406    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:23.698985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:23.698472    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:24.698101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:24.698062    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:41:24.698299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:24.698125    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 12:41:25.487617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.487578    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:41:25.493128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.493096    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.516524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.516475    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:41:25.523790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.523762    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cd1386cb-openshift-default-dockercfg-rshbn\""
Nov 02 12:41:25.583210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583178    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583227    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583249    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583497    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583564    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583814    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583887    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhx8\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.583982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.583929    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.584219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.584178    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698332    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698441    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698482    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhx8\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698519    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698585    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698643    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698700    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.698848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698731    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.699398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.698900    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.700777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.699469    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.700777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.699705    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.700777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.700370    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.707632 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.707180    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.708182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.708156    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.736849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.734491    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.740128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.738109    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.772509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.767670    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhx8\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.772509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.768227    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:25.810851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:25.810755    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:26.176022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.175655    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:41:26.179745 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:26.179700    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e748a4_4a4f_4087_a511_d6aa84662287.slice/crio-e4e51a9ce62859f456c3ab8ba4188389b58d450981ba4cf4231050e7c6c8f1f8 WatchSource:0}: Error finding container e4e51a9ce62859f456c3ab8ba4188389b58d450981ba4cf4231050e7c6c8f1f8: Status 404 returned error can't find the container with id e4e51a9ce62859f456c3ab8ba4188389b58d450981ba4cf4231050e7c6c8f1f8
Nov 02 12:41:26.183326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.183268    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:26.184486 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.183369    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:26.184486 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.183417    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:26.319098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.317428    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:26.319098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.318936    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:26.320986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.320853    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:41:26.962389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.962346    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerStarted","Data":"7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c"}
Nov 02 12:41:26.962921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:26.962412    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerStarted","Data":"e4e51a9ce62859f456c3ab8ba4188389b58d450981ba4cf4231050e7c6c8f1f8"}
Nov 02 12:41:27.039460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:27.039408    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podStartSLOduration=2.039393793 podStartE2EDuration="2.039393793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:27.022269589 +0000 UTC m=+884.671323548" watchObservedRunningTime="2026-11-02 12:41:27.039393793 +0000 UTC m=+884.688447750"
Nov 02 12:41:27.810994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:27.810955    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:27.816595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:27.816567    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:27.973326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:27.973228    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:27.975310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:27.975283    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:41:31.509127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.509092    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:41:31.513474 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.513447    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.535124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.535089    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:41:31.543245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.543219    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cd1386cb\""
Nov 02 12:41:31.543508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.543491    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cd1386cb-dockercfg-klvbg\""
Nov 02 12:41:31.595908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.595875    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.596198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.596171    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbpj\" (UniqueName: \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.698159 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.698111    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbpj\" (UniqueName: \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.698317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.698227    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.699047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.699016    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.734849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.725967    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbpj\" (UniqueName: \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:31.829988 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:31.829889    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:32.252903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:32.246615    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:41:33.173062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:33.173016    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" event={"ID":"3125fe2e-a444-4f92-a12e-4e02fac81321","Type":"ContainerStarted","Data":"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"}
Nov 02 12:41:33.174001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:33.173972    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" event={"ID":"3125fe2e-a444-4f92-a12e-4e02fac81321","Type":"ContainerStarted","Data":"02bffa4404fd622987cdab6c217a1a8b355c01d0226139c4779fb79a4f91660a"}
Nov 02 12:41:33.174242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:33.174223    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:33.335851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:33.327450    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" podStartSLOduration=2.327430558 podStartE2EDuration="2.327430558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:33.287604485 +0000 UTC m=+890.936658445" watchObservedRunningTime="2026-11-02 12:41:33.327430558 +0000 UTC m=+890.976484521"
Nov 02 12:41:34.920976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:34.920867    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:41:34.947850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:34.941097    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.128844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.122881    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203069    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203229    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203521    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203609    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203783    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.203951    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtcx\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.204026    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.204114    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.224491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.204277    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.309848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309683    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.309848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309747    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.309848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309810    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309882    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtcx\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309916    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.309957    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.310037    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.310136    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.310439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.310195    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.331959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.310738    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.331959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.311076    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.331959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.329640    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.331959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.330108    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.376848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.359941    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.376848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.361113    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.401669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.401594    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtcx\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.405847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.404242    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.406619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.406592    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.613586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.613544    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:35.976038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.976009    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:41:35.976960 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:35.976927    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe4fa08_5563_4674_976c_de81a3a58148.slice/crio-b007b4e26478eea6897cf2875a58c200ce3453836915dff1373c81b896bb77bf WatchSource:0}: Error finding container b007b4e26478eea6897cf2875a58c200ce3453836915dff1373c81b896bb77bf: Status 404 returned error can't find the container with id b007b4e26478eea6897cf2875a58c200ce3453836915dff1373c81b896bb77bf
Nov 02 12:41:35.979671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.979635    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:35.979739 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.979710    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:35.979781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:35.979739    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:36.418819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:36.417625    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" event={"ID":"7fe4fa08-5563-4674-976c-de81a3a58148","Type":"ContainerStarted","Data":"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"}
Nov 02 12:41:36.418819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:36.417696    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" event={"ID":"7fe4fa08-5563-4674-976c-de81a3a58148","Type":"ContainerStarted","Data":"b007b4e26478eea6897cf2875a58c200ce3453836915dff1373c81b896bb77bf"}
Nov 02 12:41:36.482589 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:36.482531    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podStartSLOduration=2.482513347 podStartE2EDuration="2.482513347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:36.469425758 +0000 UTC m=+894.118479718" watchObservedRunningTime="2026-11-02 12:41:36.482513347 +0000 UTC m=+894.131567351"
Nov 02 12:41:36.614304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:36.614271    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:37.615868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:37.615818    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:37.616289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:37.615887    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:38.613882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:38.613853    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:38.615954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:38.613899    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:39.614635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:39.614596    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:39.614863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:39.614670    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:39.758503 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:41:39.758456    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:41:40.614817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:40.614783    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:40.614999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:40.614865    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:41.614461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.614416    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:41.614963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.614489    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:41.655607 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.655556    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:41:41.661000 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.660975    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.661393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.661368    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:41:41.682639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.682608    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cd1386cb-serving-cert\""
Nov 02 12:41:41.850023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.849974    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btl9\" (UniqueName: \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.850359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.850332    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.853846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.850584    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.856849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.854093    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.958543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.958442    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.958728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.958550    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.958728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.958717    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btl9\" (UniqueName: \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.958893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.958794    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.960134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.960104    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.964493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.964459    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.966622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.966590    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:41.986784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:41.986754    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btl9\" (UniqueName: \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:42.282984 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.282953    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:42.614883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.614786    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:41:42.614883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.614867    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 12:41:42.648083 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:42.648048    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb95eff3_e240_4acd_8eb2_3249dad451d4.slice/crio-4091a29ae1008b4249198ae20bcb912628e866cb9b854ae1b16ff8b64d4598c2 WatchSource:0}: Error finding container 4091a29ae1008b4249198ae20bcb912628e866cb9b854ae1b16ff8b64d4598c2: Status 404 returned error can't find the container with id 4091a29ae1008b4249198ae20bcb912628e866cb9b854ae1b16ff8b64d4598c2
Nov 02 12:41:42.650567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.650543    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:41:42.655201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.655164    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:41:42.682079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.682044    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:41:42.686476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:42.686449    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:41:43.506187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.506149    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" event={"ID":"fb95eff3-e240-4acd-8eb2-3249dad451d4","Type":"ContainerStarted","Data":"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"}
Nov 02 12:41:43.506397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.506206    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" event={"ID":"fb95eff3-e240-4acd-8eb2-3249dad451d4","Type":"ContainerStarted","Data":"4091a29ae1008b4249198ae20bcb912628e866cb9b854ae1b16ff8b64d4598c2"}
Nov 02 12:41:43.506397 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.506250    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:43.554841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.554773    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" podStartSLOduration=2.175681837 podStartE2EDuration="2.554754541s" totalImagesPullingTime="379.072704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:43.55137209 +0000 UTC m=+901.200426049" watchObservedRunningTime="2026-11-02 12:41:43.554754541 +0000 UTC m=+901.203808502"
Nov 02 12:41:43.960081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.959983    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:43.960539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.960497    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:43.961411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:43.961389    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:41:44.366726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:44.366697    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:41:45.982716 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:45.982676    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:41:45.987508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:45.987482    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.023998 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.023968    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:41:46.078104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078062    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg928\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078303 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078196    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078303 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078276    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078348    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078403    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078474    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078523    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078678 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078586    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.078678 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.078619    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180192 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180154    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180220    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180280    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180310    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180350    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg928\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180484    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180593 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180576    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180641    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180655    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180726    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180727    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180770    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.180975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.180957    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.181320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.181291    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.183023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.183003    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.183184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.183160    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.205201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.205170    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.205370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.205352    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg928\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.301794 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.301761    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:46.633207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.633176    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:41:46.633380 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:46.633352    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ffd476_1f2b_4ce8_a2e5_1991d040cf45.slice/crio-56fb5379ad1516cd0a1a8bae87b0ef2fd15ff9b9cd4c971fc907781255a3ece5 WatchSource:0}: Error finding container 56fb5379ad1516cd0a1a8bae87b0ef2fd15ff9b9cd4c971fc907781255a3ece5: Status 404 returned error can't find the container with id 56fb5379ad1516cd0a1a8bae87b0ef2fd15ff9b9cd4c971fc907781255a3ece5
Nov 02 12:41:46.636003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.635967    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:46.636097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.636035    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:46.636097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:46.636062    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:41:47.564425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:47.564384    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" event={"ID":"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45","Type":"ContainerStarted","Data":"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"}
Nov 02 12:41:47.565056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:47.565031    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" event={"ID":"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45","Type":"ContainerStarted","Data":"56fb5379ad1516cd0a1a8bae87b0ef2fd15ff9b9cd4c971fc907781255a3ece5"}
Nov 02 12:41:47.660853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:47.659407    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podStartSLOduration=2.659385926 podStartE2EDuration="2.659385926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:47.64531664 +0000 UTC m=+905.294370598" watchObservedRunningTime="2026-11-02 12:41:47.659385926 +0000 UTC m=+905.308439886"
Nov 02 12:41:48.302390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:48.302338    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:48.304482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:48.304455    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:48.304606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:48.304508    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:49.303012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:49.302976    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:49.303431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:49.303031    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:49.520702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:49.520671    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:41:50.302421 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:50.302388    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:50.302619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:50.302441    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:51.302837 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:51.302800    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:51.303328 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:51.302892    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:52.020813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.020772    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:41:52.026853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.025544    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.060178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.060147    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-785a7a-serving-cert\""
Nov 02 12:41:52.070382 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.070343    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:41:52.200818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.200780    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.201008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.200945    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.201060 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.201044    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.201136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.201116    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.303058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.302961    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.304071 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.303917    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:52.304327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.304295    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:52.304892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.304058    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.305183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.305164    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.305671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.305647    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.306786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.306578    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.353848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.318394    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.384170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.383608    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.384893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.384866    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\") pod \"maas-api-e2e-api-a-785a7a-64c866d964-dlbs6\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:52.673381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:52.673291    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:53.006780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.006745    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:41:53.007966 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:41:53.007934    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc271a13b_ba95_40c5_890d_e89455096e49.slice/crio-f6da6cbe5177571f5757bd5613f9eb880f6828d813b504a357a262435b27d74e WatchSource:0}: Error finding container f6da6cbe5177571f5757bd5613f9eb880f6828d813b504a357a262435b27d74e: Status 404 returned error can't find the container with id f6da6cbe5177571f5757bd5613f9eb880f6828d813b504a357a262435b27d74e
Nov 02 12:41:53.302721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.302641    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 12:41:53.302721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.302703    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 12:41:53.643895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.640546    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" event={"ID":"c271a13b-ba95-40c5-890d-e89455096e49","Type":"ContainerStarted","Data":"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"}
Nov 02 12:41:53.643895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.640626    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" event={"ID":"c271a13b-ba95-40c5-890d-e89455096e49","Type":"ContainerStarted","Data":"f6da6cbe5177571f5757bd5613f9eb880f6828d813b504a357a262435b27d74e"}
Nov 02 12:41:53.643895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.641788    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:41:53.691170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:53.691085    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" podStartSLOduration=2.266766178 podStartE2EDuration="2.691054924s" totalImagesPullingTime="424.288746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:53.69043191 +0000 UTC m=+911.339485869" watchObservedRunningTime="2026-11-02 12:41:53.691054924 +0000 UTC m=+911.340108882"
Nov 02 12:41:54.581850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:54.581362    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:54.582068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:54.582018    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:41:54.583743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:41:54.583711    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:42:00.781424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:00.781395    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:42:01.018758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.018723    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:42:01.024902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.024871    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.060037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.051606    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:42:01.114569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.114524    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.114569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.114570    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.114815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.114779    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.114926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.114889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.114976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.114933    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.115060 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.115042    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjh2v\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.115108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.115092    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.115238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.115220    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.115377 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.115360    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.216963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.216922    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjh2v\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217171 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.216975    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217171 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217028    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217171 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217109    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217171 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217144    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217173    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217287    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217352    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217674    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217756 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217703    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217756    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.217817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.217784    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.218309 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.218286    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.220226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.220197    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.220329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.220235    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.242334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.242299    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.242672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.242647    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjh2v\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.356801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.356711    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:01.722235 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.722203    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:42:01.726196 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:01.725166    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608a9672_4410_4436_84d8_4bbecfb33439.slice/crio-f3cac6d77712b8351d524294abcf8b7d312ace0697a6c17c0234a891ed9cfc07 WatchSource:0}: Error finding container f3cac6d77712b8351d524294abcf8b7d312ace0697a6c17c0234a891ed9cfc07: Status 404 returned error can't find the container with id f3cac6d77712b8351d524294abcf8b7d312ace0697a6c17c0234a891ed9cfc07
Nov 02 12:42:01.728654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.728613    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:01.728766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.728698    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:01.728766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.728739    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:01.824320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.824284    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" event={"ID":"608a9672-4410-4436-84d8-4bbecfb33439","Type":"ContainerStarted","Data":"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"}
Nov 02 12:42:01.824760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.824353    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" event={"ID":"608a9672-4410-4436-84d8-4bbecfb33439","Type":"ContainerStarted","Data":"f3cac6d77712b8351d524294abcf8b7d312ace0697a6c17c0234a891ed9cfc07"}
Nov 02 12:42:01.879334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:01.879259    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podStartSLOduration=1.879240317 podStartE2EDuration="1.879240317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:01.872681569 +0000 UTC m=+919.521735528" watchObservedRunningTime="2026-11-02 12:42:01.879240317 +0000 UTC m=+919.528294276"
Nov 02 12:42:02.359874 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:02.358982    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:02.361525 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:02.361481    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:02.361682 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:02.361586    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:03.358141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:03.358103    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:03.358537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:03.358169    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:04.360682 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.360216    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:04.360682 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.360281    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:04.882627 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.882584    4928 generic.go:356] "Generic (PLEG): container finished" podID="82e748a4-4a4f-4087-a511-d6aa84662287" containerID="7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c" exitCode=0
Nov 02 12:42:04.882805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.882699    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerDied","Data":"7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c"}
Nov 02 12:42:04.883260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.883242    4928 scope.go:122] "RemoveContainer" containerID="7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c"
Nov 02 12:42:04.885104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.885070    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:04.885202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.885159    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:04.885269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:04.885204    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:05.357507 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:05.357469    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:05.358258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:05.357816    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:05.892358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:05.892324    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerStarted","Data":"faff8c715e8e66acfe4a9b6ea73a4a01ddd0a28b297e90514f2664c7cefef83f"}
Nov 02 12:42:06.357439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:06.357381    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:06.357908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:06.357870    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:06.811649 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:06.811623    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:42:06.811903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:06.811786    4928 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 12:42:06.812046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:06.812028    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 12:42:07.365590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:07.365543    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:07.366213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:07.366179    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:07.811792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:07.811759    4928 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 12:42:07.811986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:07.811813    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 12:42:08.357941 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:08.357902    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 12:42:08.358149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:08.357969    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 12:42:08.811377 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:08.811341    4928 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 12:42:08.811952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:08.811396    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 12:42:09.593581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:09.593552    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:09.594471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:09.594447    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:09.595440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:09.595417    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:42:09.784615 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:42:09.784584    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:42:09.811599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:09.811571    4928 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 12:42:09.812059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:09.811637    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 12:42:10.811934 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:10.811896    4928 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 12:42:10.812367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:10.811962    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 12:42:11.816020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:11.815988    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:42:11.816433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:11.816411    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:42:11.817235 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:11.817216    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:42:17.117770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.117730    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:42:17.125358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.125327    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.149707 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.149663    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:42:17.152779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.152747    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-96339b\""
Nov 02 12:42:17.156324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.156295    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:42:17.163688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.163663    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-96339b-dockercfg-vr8jq\""
Nov 02 12:42:17.197322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.197276    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.197794 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.197757    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssgdt\" (UniqueName: \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.210121 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.210095    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:42:17.210300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.210217    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.329481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.329437    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxp9\" (UniqueName: \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.330362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.330322    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.333270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.333241    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.333483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.333464    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssgdt\" (UniqueName: \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.334044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.331384    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.374895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.370696    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssgdt\" (UniqueName: \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\") pod \"payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.438858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.435624    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.438858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.435880    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxp9\" (UniqueName: \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.438858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.437361    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.451727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.451687    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:17.463166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.461937    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxp9\" (UniqueName: \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\") pod \"payload-processing-e2e-api-b-96339b-664bfbb49-hspsz\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:17.630658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:17.630561    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:18.000172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:18.000138    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:42:18.000473 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:18.000432    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b491e8_74c7_4e7e_976a_8937d93fd23e.slice/crio-1cfdba8e3bc71ae1d2e2aefb98e9683af390acb6ce3381d9834fa889539f9913 WatchSource:0}: Error finding container 1cfdba8e3bc71ae1d2e2aefb98e9683af390acb6ce3381d9834fa889539f9913: Status 404 returned error can't find the container with id 1cfdba8e3bc71ae1d2e2aefb98e9683af390acb6ce3381d9834fa889539f9913
Nov 02 12:42:18.012514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:18.012481    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:42:18.017408 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:18.017378    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc442dd11_b137_4242_a4f3_dee6f7a155e5.slice/crio-630055e3c103e94d76a93581e7c930c7a0189443019ddcde485498d18043ba81 WatchSource:0}: Error finding container 630055e3c103e94d76a93581e7c930c7a0189443019ddcde485498d18043ba81: Status 404 returned error can't find the container with id 630055e3c103e94d76a93581e7c930c7a0189443019ddcde485498d18043ba81
Nov 02 12:42:18.042658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:18.042624    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" event={"ID":"c442dd11-b137-4242-a4f3-dee6f7a155e5","Type":"ContainerStarted","Data":"630055e3c103e94d76a93581e7c930c7a0189443019ddcde485498d18043ba81"}
Nov 02 12:42:18.046768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:18.045485    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" event={"ID":"d2b491e8-74c7-4e7e-976a-8937d93fd23e","Type":"ContainerStarted","Data":"1cfdba8e3bc71ae1d2e2aefb98e9683af390acb6ce3381d9834fa889539f9913"}
Nov 02 12:42:19.074427 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.074385    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" event={"ID":"d2b491e8-74c7-4e7e-976a-8937d93fd23e","Type":"ContainerStarted","Data":"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"}
Nov 02 12:42:19.084295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.084267    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:19.105655 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.105618    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" event={"ID":"c442dd11-b137-4242-a4f3-dee6f7a155e5","Type":"ContainerStarted","Data":"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"}
Nov 02 12:42:19.106683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.106657    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:19.216849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.209253    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" podStartSLOduration=2.209233309 podStartE2EDuration="2.209233309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:19.150135491 +0000 UTC m=+936.799189450" watchObservedRunningTime="2026-11-02 12:42:19.209233309 +0000 UTC m=+936.858287269"
Nov 02 12:42:19.221811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:19.221752    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" podStartSLOduration=2.221736118 podStartE2EDuration="2.221736118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:19.217210509 +0000 UTC m=+936.866264469" watchObservedRunningTime="2026-11-02 12:42:19.221736118 +0000 UTC m=+936.870790077"
Nov 02 12:42:27.815353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.815270    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:42:27.819995 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.819974    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:27.847894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.847860    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-96339b-serving-cert\""
Nov 02 12:42:27.857585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.857540    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:42:27.952087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.952040    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:27.952267 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.952094    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxhb\" (UniqueName: \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:27.952267 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.952224    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:27.952412 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:27.952387    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.055907 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.055855    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.056090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.056038    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.056266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.056250    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.056325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.056283    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxhb\" (UniqueName: \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.057471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.057439    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.059661 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.059635    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.060393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.060312    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.081629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.081565    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxhb\" (UniqueName: \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\") pod \"maas-api-e2e-api-b-96339b-d8d6b8b88-6q574\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.138092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.138056    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:28.458180 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:28.458147    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:42:28.459678 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:28.459652    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c5fd51_42f9_48a4_940d_3502228555b4.slice/crio-7691c3fb9aa6e44bcfbf66c16c94165c7ef443b00e56e4129423bb1ca6c5f9fd WatchSource:0}: Error finding container 7691c3fb9aa6e44bcfbf66c16c94165c7ef443b00e56e4129423bb1ca6c5f9fd: Status 404 returned error can't find the container with id 7691c3fb9aa6e44bcfbf66c16c94165c7ef443b00e56e4129423bb1ca6c5f9fd
Nov 02 12:42:29.203954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:29.203507    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" event={"ID":"16c5fd51-42f9-48a4-940d-3502228555b4","Type":"ContainerStarted","Data":"1572f386973e6a6e72b64ec558ebaad9cc310e886967cd7eb7310ed7b014d839"}
Nov 02 12:42:29.203954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:29.203647    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" event={"ID":"16c5fd51-42f9-48a4-940d-3502228555b4","Type":"ContainerStarted","Data":"7691c3fb9aa6e44bcfbf66c16c94165c7ef443b00e56e4129423bb1ca6c5f9fd"}
Nov 02 12:42:29.203954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:29.203786    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:29.253030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:29.252969    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" podStartSLOduration=1.922760361 podStartE2EDuration="2.252954739s" totalImagesPullingTime="330.194378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:29.248263302 +0000 UTC m=+946.897317261" watchObservedRunningTime="2026-11-02 12:42:29.252954739 +0000 UTC m=+946.902008696"
Nov 02 12:42:31.124730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:31.124691    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:42:31.125184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:31.124766    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:42:35.291798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:35.291765    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:42:39.729814 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:42:39.729774    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:42:42.708644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:42.708607    4928 scope.go:122] "RemoveContainer" containerID="d790dfff60630baf472aea3bca9eb009d19c86e8dab2477217fa086a14877821"
Nov 02 12:42:42.719718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:42.719695    4928 scope.go:122] "RemoveContainer" containerID="27a2941f2cb5a636d45f0bdb36a39c43457fc0e6a804d33c94b0397b9ea4fbf5"
Nov 02 12:42:42.732437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:42.732413    4928 scope.go:122] "RemoveContainer" containerID="6a51337a2056212ebff54ff7d0c266a46430b97bcebca4bb5f050d998f7b1a66"
Nov 02 12:42:42.742219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:42.742194    4928 scope.go:122] "RemoveContainer" containerID="3ac6fb0525cd7d88e74344c07cfa5b4b67d8581bd7aa621dcc7926fa8c6efc63"
Nov 02 12:42:46.179354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.179320    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 12:42:46.185259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.185239    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.209450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.209416    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 12:42:46.244645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.244610    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.244645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.244651    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.244942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.244690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.244942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.244806    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.244942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.244923    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.245143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.245125    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.245237 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.245216    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.245311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.245295    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62b9\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.245367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.245327    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347302    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347540 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347384    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347540 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347432    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62b9\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347540 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347451    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347649    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347696    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347764    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347820    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347863    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.347971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.347904    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.348078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.348045    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.348257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.348232    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.348428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.348292    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.348559 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.348523    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.349910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.349885    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.350039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.350022    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.371626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.371602    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62b9\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.371940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.371920    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.504511 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.504474    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:46.862932 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:46.862896    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f3562b_200c_4e40_96ac_bc01741130b6.slice/crio-14b871caaca8a4906a5d9639c9c25b6b7d68727b18eaba295c5c76a8d0522e52 WatchSource:0}: Error finding container 14b871caaca8a4906a5d9639c9c25b6b7d68727b18eaba295c5c76a8d0522e52: Status 404 returned error can't find the container with id 14b871caaca8a4906a5d9639c9c25b6b7d68727b18eaba295c5c76a8d0522e52
Nov 02 12:42:46.865100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.865060    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 12:42:46.865736 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.865702    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:46.865862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.865768    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:46.865862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:46.865798    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:47.412010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.411974    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" event={"ID":"f9f3562b-200c-4e40-96ac-bc01741130b6","Type":"ContainerStarted","Data":"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"}
Nov 02 12:42:47.412459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.412031    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" event={"ID":"f9f3562b-200c-4e40-96ac-bc01741130b6","Type":"ContainerStarted","Data":"14b871caaca8a4906a5d9639c9c25b6b7d68727b18eaba295c5c76a8d0522e52"}
Nov 02 12:42:47.478083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.478008    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podStartSLOduration=1.477990902 podStartE2EDuration="1.477990902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:47.473112163 +0000 UTC m=+965.122166121" watchObservedRunningTime="2026-11-02 12:42:47.477990902 +0000 UTC m=+965.127044861"
Nov 02 12:42:47.504800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.504767    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:47.506306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.506276    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:47.506428 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:47.506328    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:48.504950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:48.504919    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:48.505340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:48.504971    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:49.505821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:49.505784    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:49.506233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:49.505875    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:50.504969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:50.504936    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:50.505185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:50.504990    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:51.371021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.370981    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:42:51.376994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.376965    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.389816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.389786    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:42:51.397069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.397008    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-97e08c-openshift-default-dockercfg-kscr7\""
Nov 02 12:42:51.520848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.512471    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:51.520848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.512549    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:51.548850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.545429    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.548850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.545548    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.551961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.551910    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.552188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.552168    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6lt\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.552375 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.552353    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.552565 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.552550    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.552711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.552696    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.561191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.555932    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.561481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.561449    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665579    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6lt\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665667    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665724    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665773    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665852    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665892    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.665995    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.666272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.666064    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.667864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.667741    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.669302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.668143    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.669302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.668530    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.671155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.669763    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.671155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.670027    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.671155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.670759    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.732522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.732447    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.735461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.735418    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6lt\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.735921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.735898    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:51.750153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:51.750076    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\") pod \"e2e-api-a-97e08c-openshift-default-f7c78f497-p9554\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") " pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:52.003378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.002996    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:52.412528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.412498    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:42:52.413077 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:52.413051    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba55cd_ceaf_475a_a1da_5e300c3da861.slice/crio-fa91e07a4c9f289c02054879941b9b61550a5578c3233b85c2bd477f00fe7030 WatchSource:0}: Error finding container fa91e07a4c9f289c02054879941b9b61550a5578c3233b85c2bd477f00fe7030: Status 404 returned error can't find the container with id fa91e07a4c9f289c02054879941b9b61550a5578c3233b85c2bd477f00fe7030
Nov 02 12:42:52.415780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.415748    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:52.415875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.415811    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:52.415875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.415857    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:42:52.505148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.505113    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:52.505290 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.505174    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:52.724742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.724659    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerStarted","Data":"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"}
Nov 02 12:42:52.724925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.724772    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerStarted","Data":"fa91e07a4c9f289c02054879941b9b61550a5578c3233b85c2bd477f00fe7030"}
Nov 02 12:42:52.799863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:52.799464    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podStartSLOduration=1.799447985 podStartE2EDuration="1.799447985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:52.798332225 +0000 UTC m=+970.447386183" watchObservedRunningTime="2026-11-02 12:42:52.799447985 +0000 UTC m=+970.448502011"
Nov 02 12:42:53.004882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:53.003873    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:53.006469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:53.006433    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:42:53.006726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:53.006512    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:42:53.505636 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:53.505604    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:42:53.506082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:53.505657    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 12:42:54.009501 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.009471    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:54.740158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.740123    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:54.741291 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.741268    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:42:54.780051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.780020    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:54.780685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.780661    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:54.782219 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:54.782197    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 12:42:57.559726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.559643    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:42:57.567363 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.567338    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.590686 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.587178    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-97e08c\""
Nov 02 12:42:57.591694 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.591293    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:42:57.605993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.605498    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-97e08c-dockercfg-fm4c7\""
Nov 02 12:42:57.730850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.726592    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjc8\" (UniqueName: \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.730850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.726690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.839734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.834494    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjc8\" (UniqueName: \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.839734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.834606    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.839734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.835729    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.863849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.861941    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjc8\" (UniqueName: \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\") pod \"payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:57.917136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:57.917074    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:58.244451 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:58.244414    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:42:58.245916 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:42:58.245877    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267e6707_1fa7_4998_ba4f_b747adfd2ac4.slice/crio-f804393ac842127bd1320aef3f5755edc1315c99c5c7e250d914109c3ceac21b WatchSource:0}: Error finding container f804393ac842127bd1320aef3f5755edc1315c99c5c7e250d914109c3ceac21b: Status 404 returned error can't find the container with id f804393ac842127bd1320aef3f5755edc1315c99c5c7e250d914109c3ceac21b
Nov 02 12:42:58.798595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:58.798554    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" event={"ID":"267e6707-1fa7-4998-ba4f-b747adfd2ac4","Type":"ContainerStarted","Data":"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"}
Nov 02 12:42:58.799129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:58.798616    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" event={"ID":"267e6707-1fa7-4998-ba4f-b747adfd2ac4","Type":"ContainerStarted","Data":"f804393ac842127bd1320aef3f5755edc1315c99c5c7e250d914109c3ceac21b"}
Nov 02 12:42:58.799129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:58.798642    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:42:58.843594 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:42:58.843541    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" podStartSLOduration=1.843524669 podStartE2EDuration="1.843524669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:58.842156985 +0000 UTC m=+976.491210943" watchObservedRunningTime="2026-11-02 12:42:58.843524669 +0000 UTC m=+976.492578628"
Nov 02 12:43:03.401852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.401786    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:43:03.406392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.406372    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.428520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.428488    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:43:03.429500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.429477    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9f7f01-serving-cert\""
Nov 02 12:43:03.451346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.451314    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbcd\" (UniqueName: \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.451675 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.451648    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.451954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.451934    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.452155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.452138    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.553506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.553474    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbcd\" (UniqueName: \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.553506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.553512    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.553769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.553570    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.553769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.553602    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.553769 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:03.553719    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-serving-cert: secret "maas-api-e2e-api-a-9f7f01-serving-cert" not found
Nov 02 12:43:03.553949 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:03.553801    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls podName:89d69147-8068-4256-803f-5848a3d6b9f3 nodeName:}" failed. No retries permitted until 2026-11-02 12:43:04.053780268 +0000 UTC m=+981.702834218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls") pod "maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" (UID: "89d69147-8068-4256-803f-5848a3d6b9f3") : secret "maas-api-e2e-api-a-9f7f01-serving-cert" not found
Nov 02 12:43:03.554384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.554362    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.556177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.556152    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:03.581051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:03.580987    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbcd\" (UniqueName: \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:04.062098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:04.062064    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:04.064676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:04.064645    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") pod \"maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:04.322353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:04.322269    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:04.641258 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:43:04.641222    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d69147_8068_4256_803f_5848a3d6b9f3.slice/crio-79a1b2e4d00752ee77fdce6db3ee18bbeb5af4877d33c45adcd757932ac6f3aa WatchSource:0}: Error finding container 79a1b2e4d00752ee77fdce6db3ee18bbeb5af4877d33c45adcd757932ac6f3aa: Status 404 returned error can't find the container with id 79a1b2e4d00752ee77fdce6db3ee18bbeb5af4877d33c45adcd757932ac6f3aa
Nov 02 12:43:04.642950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:04.642920    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:43:04.857602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:04.857510    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" event={"ID":"89d69147-8068-4256-803f-5848a3d6b9f3","Type":"ContainerStarted","Data":"79a1b2e4d00752ee77fdce6db3ee18bbeb5af4877d33c45adcd757932ac6f3aa"}
Nov 02 12:43:05.866512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:05.866467    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" event={"ID":"89d69147-8068-4256-803f-5848a3d6b9f3","Type":"ContainerStarted","Data":"e7e16e51471225ddd8dde93d4eca7ba348186340be8f5a1e8d7c9e95fc515e45"}
Nov 02 12:43:05.866974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:05.866536    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:05.912920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:05.912872    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" podStartSLOduration=2.569770911 podStartE2EDuration="2.912859655s" totalImagesPullingTime="343.088744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:05.912509877 +0000 UTC m=+983.561563835" watchObservedRunningTime="2026-11-02 12:43:05.912859655 +0000 UTC m=+983.561913612"
Nov 02 12:43:08.100604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.100566    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:43:08.101103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.100851    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" podUID="c271a13b-ba95-40c5-890d-e89455096e49" containerName="maas-api" containerID="cri-o://ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0" gracePeriod=30
Nov 02 12:43:08.494949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.493282    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:43:08.536340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.535869    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\") pod \"c271a13b-ba95-40c5-890d-e89455096e49\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") "
Nov 02 12:43:08.536340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.536033    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\") pod \"c271a13b-ba95-40c5-890d-e89455096e49\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") "
Nov 02 12:43:08.536340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.536082    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\") pod \"c271a13b-ba95-40c5-890d-e89455096e49\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") "
Nov 02 12:43:08.536340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.536218    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\") pod \"c271a13b-ba95-40c5-890d-e89455096e49\" (UID: \"c271a13b-ba95-40c5-890d-e89455096e49\") "
Nov 02 12:43:08.539477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.539439    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7" pod "c271a13b-ba95-40c5-890d-e89455096e49" (UID: "c271a13b-ba95-40c5-890d-e89455096e49"). InnerVolumeSpecName "kube-api-access-897k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:08.540009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.539783    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs" pod "c271a13b-ba95-40c5-890d-e89455096e49" (UID: "c271a13b-ba95-40c5-890d-e89455096e49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:43:08.549114 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.549064    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca" pod "c271a13b-ba95-40c5-890d-e89455096e49" (UID: "c271a13b-ba95-40c5-890d-e89455096e49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:08.565074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.564943    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls" pod "c271a13b-ba95-40c5-890d-e89455096e49" (UID: "c271a13b-ba95-40c5-890d-e89455096e49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:43:08.642848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.642510    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:08.642848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.642556    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897k7\" (UniqueName: \"kubernetes.io/projected/c271a13b-ba95-40c5-890d-e89455096e49-kube-api-access-897k7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:08.642848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.642571    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c271a13b-ba95-40c5-890d-e89455096e49-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:08.642848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.642585    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c271a13b-ba95-40c5-890d-e89455096e49-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:08.906345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.906288    4928 generic.go:356] "Generic (PLEG): container finished" podID="c271a13b-ba95-40c5-890d-e89455096e49" containerID="ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0" exitCode=0
Nov 02 12:43:08.906567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.906351    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" event={"ID":"c271a13b-ba95-40c5-890d-e89455096e49","Type":"ContainerDied","Data":"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"}
Nov 02 12:43:08.906567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.906464    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6" event={"ID":"c271a13b-ba95-40c5-890d-e89455096e49","Type":"ContainerDied","Data":"f6da6cbe5177571f5757bd5613f9eb880f6828d813b504a357a262435b27d74e"}
Nov 02 12:43:08.906567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.906497    4928 scope.go:122] "RemoveContainer" containerID="ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"
Nov 02 12:43:08.906867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.906848    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"
Nov 02 12:43:08.927294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.927268    4928 scope.go:122] "RemoveContainer" containerID="ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"
Nov 02 12:43:08.927730 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:08.927695    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0\": container with ID starting with ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0 not found: ID does not exist" containerID="ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"
Nov 02 12:43:08.927901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.927747    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0"} err="failed to get container status \"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0\": rpc error: code = NotFound desc = could not find container \"ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0\": container with ID starting with ef73228c795c512aeaec7aac93ce9e207be97d508dde9233cf11bbff64ffb2d0 not found: ID does not exist"
Nov 02 12:43:08.969921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.969886    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:43:08.973756 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:08.973728    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785a7a-64c866d964-dlbs6"]
Nov 02 12:43:09.730135 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:09.730099    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:43:09.810090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:09.810056    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:43:10.522280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:10.522239    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c271a13b-ba95-40c5-890d-e89455096e49" path="/var/lib/kubelet/pods/c271a13b-ba95-40c5-890d-e89455096e49/volumes"
Nov 02 12:43:11.889213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.889180    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:43:11.990379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.990341    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:43:11.991521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.991494    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c271a13b-ba95-40c5-890d-e89455096e49" containerName="maas-api"
Nov 02 12:43:11.991683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.991665    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c271a13b-ba95-40c5-890d-e89455096e49" containerName="maas-api"
Nov 02 12:43:11.991875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.991860    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c271a13b-ba95-40c5-890d-e89455096e49" containerName="maas-api"
Nov 02 12:43:11.997606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:11.997573    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.027849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.015375    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-97e08c-serving-cert\""
Nov 02 12:43:12.027849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.025416    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:43:12.119726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.119688    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.119942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.119737    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxwd\" (UniqueName: \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.119942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.119803    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.120061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.120016    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.222048 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.221965    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.222048 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.222019    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxwd\" (UniqueName: \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.222252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.222051    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.222252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.222213    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.222915 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.222883    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.225496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.225471    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.253738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.253701    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxwd\" (UniqueName: \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.259630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.259601    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\") pod \"maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.333781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.333738    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:12.659963 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:43:12.659920    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746c6062_3cdf_4f3d_9dd0_b0fdc3929c71.slice/crio-dd304ce51e140aed3df2d49ca72369aec9562b3143f3b929ea841bf6360d07e1 WatchSource:0}: Error finding container dd304ce51e140aed3df2d49ca72369aec9562b3143f3b929ea841bf6360d07e1: Status 404 returned error can't find the container with id dd304ce51e140aed3df2d49ca72369aec9562b3143f3b929ea841bf6360d07e1
Nov 02 12:43:12.660539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.660503    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:43:12.949723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:12.949634    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" event={"ID":"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71","Type":"ContainerStarted","Data":"dd304ce51e140aed3df2d49ca72369aec9562b3143f3b929ea841bf6360d07e1"}
Nov 02 12:43:13.985038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:13.984992    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" event={"ID":"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71","Type":"ContainerStarted","Data":"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"}
Nov 02 12:43:13.985808 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:13.985787    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:14.070852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:14.062050    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" podStartSLOduration=2.743582231 podStartE2EDuration="3.06202941s" totalImagesPullingTime="318.447179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:14.034644605 +0000 UTC m=+991.683698563" watchObservedRunningTime="2026-11-02 12:43:14.06202941 +0000 UTC m=+991.711083367"
Nov 02 12:43:19.909843 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:19.907926    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:43:20.016050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:20.015697    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:43:21.019514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:21.019480    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:43:24.908850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:24.908787    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" podUID="c442dd11-b137-4242-a4f3-dee6f7a155e5" containerName="payload-processing" containerID="cri-o://3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca" gracePeriod=25
Nov 02 12:43:25.016967 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.016923    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" podUID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" containerName="payload-pre-processing" containerID="cri-o://0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f" gracePeriod=25
Nov 02 12:43:25.248305 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.246778    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:43:25.378172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.378145    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:43:25.417014 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.416980    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\") pod \"c442dd11-b137-4242-a4f3-dee6f7a155e5\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") "
Nov 02 12:43:25.417191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.417112    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\" (UniqueName: \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\") pod \"c442dd11-b137-4242-a4f3-dee6f7a155e5\" (UID: \"c442dd11-b137-4242-a4f3-dee6f7a155e5\") "
Nov 02 12:43:25.417291 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.417267    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume" pod "c442dd11-b137-4242-a4f3-dee6f7a155e5" (UID: "c442dd11-b137-4242-a4f3-dee6f7a155e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:25.417685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.417660    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c442dd11-b137-4242-a4f3-dee6f7a155e5-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:25.419224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.419203    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9" pod "c442dd11-b137-4242-a4f3-dee6f7a155e5" (UID: "c442dd11-b137-4242-a4f3-dee6f7a155e5"). InnerVolumeSpecName "kube-api-access-ksxp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:25.500355 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.500318    4928 generic.go:356] "Generic (PLEG): container finished" podID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" containerID="0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f" exitCode=0
Nov 02 12:43:25.500537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.500399    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"
Nov 02 12:43:25.500537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.500412    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" event={"ID":"d2b491e8-74c7-4e7e-976a-8937d93fd23e","Type":"ContainerDied","Data":"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"}
Nov 02 12:43:25.500537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.500473    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk" event={"ID":"d2b491e8-74c7-4e7e-976a-8937d93fd23e","Type":"ContainerDied","Data":"1cfdba8e3bc71ae1d2e2aefb98e9683af390acb6ce3381d9834fa889539f9913"}
Nov 02 12:43:25.500537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.500513    4928 scope.go:122] "RemoveContainer" containerID="0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"
Nov 02 12:43:25.502292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.502270    4928 generic.go:356] "Generic (PLEG): container finished" podID="c442dd11-b137-4242-a4f3-dee6f7a155e5" containerID="3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca" exitCode=0
Nov 02 12:43:25.502396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.502327    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"
Nov 02 12:43:25.502396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.502360    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" event={"ID":"c442dd11-b137-4242-a4f3-dee6f7a155e5","Type":"ContainerDied","Data":"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"}
Nov 02 12:43:25.502492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.502410    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz" event={"ID":"c442dd11-b137-4242-a4f3-dee6f7a155e5","Type":"ContainerDied","Data":"630055e3c103e94d76a93581e7c930c7a0189443019ddcde485498d18043ba81"}
Nov 02 12:43:25.515244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.515223    4928 scope.go:122] "RemoveContainer" containerID="0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"
Nov 02 12:43:25.515813 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:25.515790    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f\": container with ID starting with 0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f not found: ID does not exist" containerID="0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"
Nov 02 12:43:25.515939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.515818    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f"} err="failed to get container status \"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f\": rpc error: code = NotFound desc = could not find container \"0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f\": container with ID starting with 0e71819198333cbb0928d0dc058f6738ff6fd2266a08938e21d29fa64543c11f not found: ID does not exist"
Nov 02 12:43:25.515939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.515853    4928 scope.go:122] "RemoveContainer" containerID="3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"
Nov 02 12:43:25.518530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.518506    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\" (UniqueName: \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\") pod \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") "
Nov 02 12:43:25.518716 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.518698    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\") pod \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\" (UID: \"d2b491e8-74c7-4e7e-976a-8937d93fd23e\") "
Nov 02 12:43:25.519075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.519050    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume" pod "d2b491e8-74c7-4e7e-976a-8937d93fd23e" (UID: "d2b491e8-74c7-4e7e-976a-8937d93fd23e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:25.519325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.519303    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxp9\" (UniqueName: \"kubernetes.io/projected/c442dd11-b137-4242-a4f3-dee6f7a155e5-kube-api-access-ksxp9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:25.519399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.519330    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b491e8-74c7-4e7e-976a-8937d93fd23e-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:25.521070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.521046    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt" pod "d2b491e8-74c7-4e7e-976a-8937d93fd23e" (UID: "d2b491e8-74c7-4e7e-976a-8937d93fd23e"). InnerVolumeSpecName "kube-api-access-ssgdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:25.526506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.526492    4928 scope.go:122] "RemoveContainer" containerID="3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"
Nov 02 12:43:25.526782 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:25.526764    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca\": container with ID starting with 3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca not found: ID does not exist" containerID="3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"
Nov 02 12:43:25.526867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.526788    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca"} err="failed to get container status \"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca\": rpc error: code = NotFound desc = could not find container \"3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca\": container with ID starting with 3af806da72ab56bab6ca9e7977cf903c7ef28080b55a53eb09752c03c3d11dca not found: ID does not exist"
Nov 02 12:43:25.559402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.559368    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:43:25.563414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.563389    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96339b-664bfbb49-hspsz"]
Nov 02 12:43:25.621347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.621316    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssgdt\" (UniqueName: \"kubernetes.io/projected/d2b491e8-74c7-4e7e-976a-8937d93fd23e-kube-api-access-ssgdt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:25.851275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.851239    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:43:25.858980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:25.858941    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96339b-696f77c85f-2mlfk"]
Nov 02 12:43:26.499175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:26.499141    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c442dd11-b137-4242-a4f3-dee6f7a155e5" path="/var/lib/kubelet/pods/c442dd11-b137-4242-a4f3-dee6f7a155e5/volumes"
Nov 02 12:43:26.499627 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:26.499612    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" path="/var/lib/kubelet/pods/d2b491e8-74c7-4e7e-976a-8937d93fd23e/volumes"
Nov 02 12:43:28.045518 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.045480    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:43:28.046446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046421    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c442dd11-b137-4242-a4f3-dee6f7a155e5" containerName="payload-processing"
Nov 02 12:43:28.046446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046444    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c442dd11-b137-4242-a4f3-dee6f7a155e5" containerName="payload-processing"
Nov 02 12:43:28.046590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046487    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" containerName="payload-pre-processing"
Nov 02 12:43:28.046590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046492    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" containerName="payload-pre-processing"
Nov 02 12:43:28.046668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046595    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b491e8-74c7-4e7e-976a-8937d93fd23e" containerName="payload-pre-processing"
Nov 02 12:43:28.046668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.046607    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c442dd11-b137-4242-a4f3-dee6f7a155e5" containerName="payload-processing"
Nov 02 12:43:28.096229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.096196    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.101840 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.101799    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:43:28.168516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168466    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.168729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168697    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.168802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168737    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.168802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168781    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.168939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168818    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt9v\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.168939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168910    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.169050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168943    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.169050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.168994    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.169159 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.169057    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.171254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.171223    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9f7f01-openshift-default-dockercfg-hkzth\""
Nov 02 12:43:28.270467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270416    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270462    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270590    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270639    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt9v\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270720    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270753    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.270857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270797    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.270875    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271003    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271085    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271323    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271408    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271511 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271474    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.271566 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.271521    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.273032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.273002    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.273455 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.273437    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.295652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.295589    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.295801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.295649    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt9v\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.417993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.417960    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:28.779352 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:43:28.779315    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b0b05f_a3e3_48bf_92dd_e7e6fa0f4ae6.slice/crio-ac49a43b37a3931415f5a7207cf267b096e94bd0ef815b8d1ab0331359720973 WatchSource:0}: Error finding container ac49a43b37a3931415f5a7207cf267b096e94bd0ef815b8d1ab0331359720973: Status 404 returned error can't find the container with id ac49a43b37a3931415f5a7207cf267b096e94bd0ef815b8d1ab0331359720973
Nov 02 12:43:28.780928 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.780897    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:43:28.783040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.783003    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:28.783142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.783086    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:28.783207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:28.783139    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:29.565143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:29.565069    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" event={"ID":"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6","Type":"ContainerStarted","Data":"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"}
Nov 02 12:43:29.565143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:29.565139    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" event={"ID":"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6","Type":"ContainerStarted","Data":"ac49a43b37a3931415f5a7207cf267b096e94bd0ef815b8d1ab0331359720973"}
Nov 02 12:43:29.659855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:29.658960    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podStartSLOduration=1.658941886 podStartE2EDuration="1.658941886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:29.641952079 +0000 UTC m=+1007.291006369" watchObservedRunningTime="2026-11-02 12:43:29.658941886 +0000 UTC m=+1007.307995905"
Nov 02 12:43:30.418845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:30.418790    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:30.420336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:30.420309    4928 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 12:43:30.420453 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:30.420364    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 12:43:31.419925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:31.419888    4928 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 12:43:31.420324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:31.419940    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 12:43:32.418334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:32.418295    4928 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 12:43:32.418520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:32.418382    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 12:43:33.418869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:33.418816    4928 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 12:43:33.419276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:33.418897    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 12:43:34.418599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:34.418557    4928 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 12:43:34.418799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:34.418624    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 12:43:35.255877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.254169    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:43:35.255877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.254426    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" podUID="16c5fd51-42f9-48a4-940d-3502228555b4" containerName="maas-api" containerID="cri-o://1572f386973e6a6e72b64ec558ebaad9cc310e886967cd7eb7310ed7b014d839" gracePeriod=30
Nov 02 12:43:35.423636 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.423602    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:35.424047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.424021    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:35.424719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.424696    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:43:35.654903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.647859    4928 generic.go:356] "Generic (PLEG): container finished" podID="16c5fd51-42f9-48a4-940d-3502228555b4" containerID="1572f386973e6a6e72b64ec558ebaad9cc310e886967cd7eb7310ed7b014d839" exitCode=0
Nov 02 12:43:35.654903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.649150    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" event={"ID":"16c5fd51-42f9-48a4-940d-3502228555b4","Type":"ContainerDied","Data":"1572f386973e6a6e72b64ec558ebaad9cc310e886967cd7eb7310ed7b014d839"}
Nov 02 12:43:35.848263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.848212    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:43:35.948939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.945366    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\") pod \"16c5fd51-42f9-48a4-940d-3502228555b4\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") "
Nov 02 12:43:35.948939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.945420    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\") pod \"16c5fd51-42f9-48a4-940d-3502228555b4\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") "
Nov 02 12:43:35.948939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.945627    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\") pod \"16c5fd51-42f9-48a4-940d-3502228555b4\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") "
Nov 02 12:43:35.948939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.945727    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\" (UniqueName: \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\") pod \"16c5fd51-42f9-48a4-940d-3502228555b4\" (UID: \"16c5fd51-42f9-48a4-940d-3502228555b4\") "
Nov 02 12:43:35.956912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.954511    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb" pod "16c5fd51-42f9-48a4-940d-3502228555b4" (UID: "16c5fd51-42f9-48a4-940d-3502228555b4"). InnerVolumeSpecName "kube-api-access-djxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:35.979245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.973323    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca" pod "16c5fd51-42f9-48a4-940d-3502228555b4" (UID: "16c5fd51-42f9-48a4-940d-3502228555b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:35.982985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.981054    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls" pod "16c5fd51-42f9-48a4-940d-3502228555b4" (UID: "16c5fd51-42f9-48a4-940d-3502228555b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:43:35.986256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:35.986218    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs" pod "16c5fd51-42f9-48a4-940d-3502228555b4" (UID: "16c5fd51-42f9-48a4-940d-3502228555b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:43:36.054274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.054241    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5fd51-42f9-48a4-940d-3502228555b4-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:36.054274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.054268    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxhb\" (UniqueName: \"kubernetes.io/projected/16c5fd51-42f9-48a4-940d-3502228555b4-kube-api-access-djxhb\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:36.054274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.054278    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:36.054602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.054286    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c5fd51-42f9-48a4-940d-3502228555b4-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:43:36.658916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.658834    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"
Nov 02 12:43:36.658916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.658854    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574" event={"ID":"16c5fd51-42f9-48a4-940d-3502228555b4","Type":"ContainerDied","Data":"7691c3fb9aa6e44bcfbf66c16c94165c7ef443b00e56e4129423bb1ca6c5f9fd"}
Nov 02 12:43:36.659405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.658927    4928 scope.go:122] "RemoveContainer" containerID="1572f386973e6a6e72b64ec558ebaad9cc310e886967cd7eb7310ed7b014d839"
Nov 02 12:43:36.717097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.717071    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:43:36.732532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:36.732503    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96339b-d8d6b8b88-6q574"]
Nov 02 12:43:38.499435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:38.499400    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c5fd51-42f9-48a4-940d-3502228555b4" path="/var/lib/kubelet/pods/16c5fd51-42f9-48a4-940d-3502228555b4/volumes"
Nov 02 12:43:39.730985 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:43:39.730947    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:43:44.166203 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.166149    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:43:44.167447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.167419    4928 cpu_manager.go:414] "containerMap: removing container" podUID="16c5fd51-42f9-48a4-940d-3502228555b4" containerName="maas-api"
Nov 02 12:43:44.167535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.167450    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c5fd51-42f9-48a4-940d-3502228555b4" containerName="maas-api"
Nov 02 12:43:44.167760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.167744    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c5fd51-42f9-48a4-940d-3502228555b4" containerName="maas-api"
Nov 02 12:43:44.176066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.174430    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.178234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.177921    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:43:44.190572 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.190541    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6a3af1-dockercfg-tz7tw\""
Nov 02 12:43:44.204047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.204027    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6a3af1\""
Nov 02 12:43:44.222890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.222864    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:43:44.258358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.258320    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.278323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.278281    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:43:44.510614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.510576    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjvc\" (UniqueName: \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.512246 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.512211    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.639901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.639855    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfst\" (UniqueName: \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.640358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.640329    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.640556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.640538    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.655850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.640742    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjvc\" (UniqueName: \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.656287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.649770    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.676595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.676420    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjvc\" (UniqueName: \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\") pod \"payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.758141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.758093    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.758448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.758428    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfst\" (UniqueName: \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.760425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.760394    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.784590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.784523    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfst\" (UniqueName: \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\") pod \"payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:44.796203 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.796130    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:44.909326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:44.907899    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:45.185071 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.184642    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:43:45.305172 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:43:45.305146    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5271259_bfa2_4f75_8fab_cf28eca3455b.slice/crio-41b37191072155079c57f016ec887369a3c49f4470daed83d85d16228d6aea8d WatchSource:0}: Error finding container 41b37191072155079c57f016ec887369a3c49f4470daed83d85d16228d6aea8d: Status 404 returned error can't find the container with id 41b37191072155079c57f016ec887369a3c49f4470daed83d85d16228d6aea8d
Nov 02 12:43:45.305736 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.305715    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.798697    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" event={"ID":"b5271259-bfa2-4f75-8fab-cf28eca3455b","Type":"ContainerStarted","Data":"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"}
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.798778    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.798803    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" event={"ID":"b5271259-bfa2-4f75-8fab-cf28eca3455b","Type":"ContainerStarted","Data":"41b37191072155079c57f016ec887369a3c49f4470daed83d85d16228d6aea8d"}
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.801006    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" event={"ID":"2082d000-5279-4b2b-bc0a-d4e23a9da359","Type":"ContainerStarted","Data":"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"}
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.801045    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" event={"ID":"2082d000-5279-4b2b-bc0a-d4e23a9da359","Type":"ContainerStarted","Data":"35e1363fdaed7286f5de144dd34853190d4f4552008f5c7f11e3abcee6c3330b"}
Nov 02 12:43:45.802141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.801592    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:45.864537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.864465    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" podStartSLOduration=1.864445471 podStartE2EDuration="1.864445471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:45.846232045 +0000 UTC m=+1023.495286004" watchObservedRunningTime="2026-11-02 12:43:45.864445471 +0000 UTC m=+1023.513499430"
Nov 02 12:43:45.893136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:45.893081    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" podStartSLOduration=1.893066477 podStartE2EDuration="1.893066477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:45.88997935 +0000 UTC m=+1023.539033308" watchObservedRunningTime="2026-11-02 12:43:45.893066477 +0000 UTC m=+1023.542120472"
Nov 02 12:43:47.807621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:47.807572    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:43:50.559994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.559953    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:43:50.567249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.567221    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.584743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.584618    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c79cde-serving-cert\""
Nov 02 12:43:50.607621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.607582    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:43:50.740691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.740650    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th24b\" (UniqueName: \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.740910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.740698    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.740910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.740868    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.741042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.740954    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.842930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.842609    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.842930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.842758    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.844111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.843361    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th24b\" (UniqueName: \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.844111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.843412    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.844111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.843453    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.846761 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.846725    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.846952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.846872    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.874604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.874564    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th24b\" (UniqueName: \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\") pod \"maas-api-e2e-api-b-c79cde-58d456b7-5jjks\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:50.891884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:50.891591    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:51.233737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:51.233706    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:43:51.234355 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:43:51.234327    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f84930_febf_40ed_a33b_6b0120b71e4c.slice/crio-c6555f7b305c5f622d30bbc79c7cf76775c3bfb42b7dd399396df7044f038d95 WatchSource:0}: Error finding container c6555f7b305c5f622d30bbc79c7cf76775c3bfb42b7dd399396df7044f038d95: Status 404 returned error can't find the container with id c6555f7b305c5f622d30bbc79c7cf76775c3bfb42b7dd399396df7044f038d95
Nov 02 12:43:51.895902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:51.895754    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" event={"ID":"74f84930-febf-40ed-a33b-6b0120b71e4c","Type":"ContainerStarted","Data":"c6555f7b305c5f622d30bbc79c7cf76775c3bfb42b7dd399396df7044f038d95"}
Nov 02 12:43:51.896283 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:51.895959    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:51.948697 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:51.948638    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" podStartSLOduration=1.458268189 podStartE2EDuration="1.948619122s" totalImagesPullingTime="490.350933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:51.948119457 +0000 UTC m=+1029.597173415" watchObservedRunningTime="2026-11-02 12:43:51.948619122 +0000 UTC m=+1029.597673080"
Nov 02 12:43:52.808182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:52.808130    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing" containerID="cri-o://32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22" gracePeriod=25
Nov 02 12:43:52.905923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:52.905886    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" event={"ID":"74f84930-febf-40ed-a33b-6b0120b71e4c","Type":"ContainerStarted","Data":"d683f04fe6448be5a787e0b3d9c9161ff3434c28657140c3dcb17dc02da062b4"}
Nov 02 12:43:53.927065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.926959    4928 generic.go:356] "Generic (PLEG): container finished" podID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerID="eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab" exitCode=0
Nov 02 12:43:53.927065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.927004    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerDied","Data":"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"}
Nov 02 12:43:53.927673 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.927538    4928 scope.go:122] "RemoveContainer" containerID="eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"
Nov 02 12:43:53.929154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.929120    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:53.929233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.929206    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:53.929274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:53.929254    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:43:54.365758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:54.365722    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:54.745847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:54.744942    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:43:54.959272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:54.959233    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerStarted","Data":"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"}
Nov 02 12:43:55.004248 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:55.004159    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:43:55.005351 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:55.005301    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:43:55.005484 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:55.005369    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:43:56.004384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:56.004343    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:43:56.004806 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:56.004410    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:43:56.811691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:56.811657    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:43:57.003852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:57.003791    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:43:57.004057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:57.003876    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:43:57.819992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:57.819949    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:43:58.003855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:58.003801    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:43:58.004028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:58.003877    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:43:58.961355 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:58.961318    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:43:59.003591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:59.003560    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:43:59.003760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:43:59.003609    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:44:00.007166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:00.006304    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 12:44:00.007166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:00.006390    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 12:44:01.007949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:01.007913    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:44:01.008468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:01.008443    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:44:01.009193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:01.009173    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:44:02.380008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.379967    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"]
Nov 02 12:44:02.385545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.385505    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.402221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.402195    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6fgxx\""
Nov 02 12:44:02.404614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.404584    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 12:44:02.424144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.424117    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"]
Nov 02 12:44:02.458705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.458647    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.458705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.458742    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdks\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-kube-api-access-hzdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.458705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.458801    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.459124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.459036    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.459124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.459107    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cc2c54-2032-4ccf-a958-af2200e3d477-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.459228 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.459208    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.459285 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.459254    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.502341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.502311    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 12:44:02.502341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.502334    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 12:44:02.560611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.560567    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.560788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.560630    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cc2c54-2032-4ccf-a958-af2200e3d477-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.560788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.560696    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.560788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.560730    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.561093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.560822    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.561152 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.561118    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdks\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-kube-api-access-hzdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.561202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.561166    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.561717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.561660    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.563741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.563715    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cc2c54-2032-4ccf-a958-af2200e3d477-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.563933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.563910    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.564162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.564137    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.564257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.564233    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.586348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.586316    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.586506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.586489    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdks\" (UniqueName: \"kubernetes.io/projected/a2cc2c54-2032-4ccf-a958-af2200e3d477-kube-api-access-hzdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cznj4\" (UID: \"a2cc2c54-2032-4ccf-a958-af2200e3d477\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:02.699488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:02.699399    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:03.030959 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:44:03.030925    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cc2c54_2032_4ccf_a958_af2200e3d477.slice/crio-550f26703539f3508953213f10c16202fe362173626c5750031fab93e61c0acb WatchSource:0}: Error finding container 550f26703539f3508953213f10c16202fe362173626c5750031fab93e61c0acb: Status 404 returned error can't find the container with id 550f26703539f3508953213f10c16202fe362173626c5750031fab93e61c0acb
Nov 02 12:44:03.037541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:03.037510    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"]
Nov 02 12:44:03.074635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:03.074596    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4" event={"ID":"a2cc2c54-2032-4ccf-a958-af2200e3d477","Type":"ContainerStarted","Data":"550f26703539f3508953213f10c16202fe362173626c5750031fab93e61c0acb"}
Nov 02 12:44:04.365405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:04.365362    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:05.919370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:05.919331    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:44:05.919709 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:05.919403    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:44:06.102130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:06.102092    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4" event={"ID":"a2cc2c54-2032-4ccf-a958-af2200e3d477","Type":"ContainerStarted","Data":"c7b9aa71175f109cbe8596fca5372c1bacee3ac25d1e898c869e8a9fe7a5330d"}
Nov 02 12:44:06.102679 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:06.102158    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:06.103889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:06.103862    4928 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cznj4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:44:06.104026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:06.103916    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4" podUID="a2cc2c54-2032-4ccf-a958-af2200e3d477" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:44:06.170663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:06.170549    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4" podStartSLOduration=1.287074303 podStartE2EDuration="4.170534082s" totalImagesPullingTime="2.883459779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:06.166265311 +0000 UTC m=+1043.815319274" watchObservedRunningTime="2026-11-02 12:44:06.170534082 +0000 UTC m=+1043.819588040"
Nov 02 12:44:07.132858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:07.131090    4928 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cznj4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:44:07.132858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:07.131146    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4" podUID="a2cc2c54-2032-4ccf-a958-af2200e3d477" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:44:09.737707 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:09.737649    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:44:10.131970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:10.131877    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cznj4"
Nov 02 12:44:13.107127 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.107098    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:44:13.157423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.157387    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\") pod \"3125fe2e-a444-4f92-a12e-4e02fac81321\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") "
Nov 02 12:44:13.157617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.157603    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\" (UniqueName: \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\") pod \"3125fe2e-a444-4f92-a12e-4e02fac81321\" (UID: \"3125fe2e-a444-4f92-a12e-4e02fac81321\") "
Nov 02 12:44:13.157882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.157815    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume" pod "3125fe2e-a444-4f92-a12e-4e02fac81321" (UID: "3125fe2e-a444-4f92-a12e-4e02fac81321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:13.158276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.158255    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3125fe2e-a444-4f92-a12e-4e02fac81321-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:13.159916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.159884    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj" pod "3125fe2e-a444-4f92-a12e-4e02fac81321" (UID: "3125fe2e-a444-4f92-a12e-4e02fac81321"). InnerVolumeSpecName "kube-api-access-lmbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:13.188979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.188945    4928 generic.go:356] "Generic (PLEG): container finished" podID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerID="32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22" exitCode=0
Nov 02 12:44:13.189110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.188998    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"
Nov 02 12:44:13.189110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.189026    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" event={"ID":"3125fe2e-a444-4f92-a12e-4e02fac81321","Type":"ContainerDied","Data":"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"}
Nov 02 12:44:13.189110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.189083    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b" event={"ID":"3125fe2e-a444-4f92-a12e-4e02fac81321","Type":"ContainerDied","Data":"02bffa4404fd622987cdab6c217a1a8b355c01d0226139c4779fb79a4f91660a"}
Nov 02 12:44:13.189110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.189111    4928 scope.go:122] "RemoveContainer" containerID="32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"
Nov 02 12:44:13.202802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.202778    4928 scope.go:122] "RemoveContainer" containerID="32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"
Nov 02 12:44:13.203229 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:13.203199    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22\": container with ID starting with 32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22 not found: ID does not exist" containerID="32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"
Nov 02 12:44:13.203327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.203240    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22"} err="failed to get container status \"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22\": rpc error: code = NotFound desc = could not find container \"32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22\": container with ID starting with 32283f0a0d86862c41ca34b9cdf4bec7b3a2a7f558904435d2c927520210ca22 not found: ID does not exist"
Nov 02 12:44:13.251368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.251341    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:44:13.256594 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.256569    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-f679b"]
Nov 02 12:44:13.259429 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:13.259402    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbpj\" (UniqueName: \"kubernetes.io/projected/3125fe2e-a444-4f92-a12e-4e02fac81321-kube-api-access-lmbpj\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:14.504557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:14.504510    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" path="/var/lib/kubelet/pods/3125fe2e-a444-4f92-a12e-4e02fac81321/volumes"
Nov 02 12:44:23.391852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:23.390933    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:23.705063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:23.704949    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:26.686021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:26.685982    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:44:26.686515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:26.686253    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" podUID="fb95eff3-e240-4acd-8eb2-3249dad451d4" containerName="maas-api" containerID="cri-o://6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe" gracePeriod=30
Nov 02 12:44:27.110792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.110753    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:44:27.183402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.168745    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\") pod \"fb95eff3-e240-4acd-8eb2-3249dad451d4\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") "
Nov 02 12:44:27.192114 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.192063    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\" (UniqueName: \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\") pod \"fb95eff3-e240-4acd-8eb2-3249dad451d4\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") "
Nov 02 12:44:27.192594 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.192576    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\") pod \"fb95eff3-e240-4acd-8eb2-3249dad451d4\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") "
Nov 02 12:44:27.193081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.193060    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\") pod \"fb95eff3-e240-4acd-8eb2-3249dad451d4\" (UID: \"fb95eff3-e240-4acd-8eb2-3249dad451d4\") "
Nov 02 12:44:27.195649 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.195625    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca" pod "fb95eff3-e240-4acd-8eb2-3249dad451d4" (UID: "fb95eff3-e240-4acd-8eb2-3249dad451d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:27.233121 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.233077    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls" pod "fb95eff3-e240-4acd-8eb2-3249dad451d4" (UID: "fb95eff3-e240-4acd-8eb2-3249dad451d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:27.243021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.242954    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs" pod "fb95eff3-e240-4acd-8eb2-3249dad451d4" (UID: "fb95eff3-e240-4acd-8eb2-3249dad451d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:27.257846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.253973    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9" pod "fb95eff3-e240-4acd-8eb2-3249dad451d4" (UID: "fb95eff3-e240-4acd-8eb2-3249dad451d4"). InnerVolumeSpecName "kube-api-access-4btl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:27.303620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.303584    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.303889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.303868    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btl9\" (UniqueName: \"kubernetes.io/projected/fb95eff3-e240-4acd-8eb2-3249dad451d4-kube-api-access-4btl9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.304007 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.303993    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb95eff3-e240-4acd-8eb2-3249dad451d4-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.304098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.304086    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb95eff3-e240-4acd-8eb2-3249dad451d4-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.742346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.742295    4928 generic.go:356] "Generic (PLEG): container finished" podID="fb95eff3-e240-4acd-8eb2-3249dad451d4" containerID="6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe" exitCode=0
Nov 02 12:44:27.742899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.742376    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"
Nov 02 12:44:27.742899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.742384    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" event={"ID":"fb95eff3-e240-4acd-8eb2-3249dad451d4","Type":"ContainerDied","Data":"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"}
Nov 02 12:44:27.742899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.742462    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w" event={"ID":"fb95eff3-e240-4acd-8eb2-3249dad451d4","Type":"ContainerDied","Data":"4091a29ae1008b4249198ae20bcb912628e866cb9b854ae1b16ff8b64d4598c2"}
Nov 02 12:44:27.742899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.742489    4928 scope.go:122] "RemoveContainer" containerID="6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"
Nov 02 12:44:27.761024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.761001    4928 scope.go:122] "RemoveContainer" containerID="6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"
Nov 02 12:44:27.761383 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:27.761356    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe\": container with ID starting with 6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe not found: ID does not exist" containerID="6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"
Nov 02 12:44:27.761449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.761405    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe"} err="failed to get container status \"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe\": rpc error: code = NotFound desc = could not find container \"6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe\": container with ID starting with 6ff5b5039b7bd74159126e9f0aa1961cdbe10f46a7f4bf360e25eec8cd2872fe not found: ID does not exist"
Nov 02 12:44:27.805677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.805638    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:44:27.821334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:27.821299    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-xs78w"]
Nov 02 12:44:28.499708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:28.499676    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb95eff3-e240-4acd-8eb2-3249dad451d4" path="/var/lib/kubelet/pods/fb95eff3-e240-4acd-8eb2-3249dad451d4/volumes"
Nov 02 12:44:30.151187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.151154    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:44:30.501873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.501838    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:44:30.502109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.502057    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" podUID="89d69147-8068-4256-803f-5848a3d6b9f3" containerName="maas-api" containerID="cri-o://e7e16e51471225ddd8dde93d4eca7ba348186340be8f5a1e8d7c9e95fc515e45" gracePeriod=30
Nov 02 12:44:30.777792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.777706    4928 generic.go:356] "Generic (PLEG): container finished" podID="89d69147-8068-4256-803f-5848a3d6b9f3" containerID="e7e16e51471225ddd8dde93d4eca7ba348186340be8f5a1e8d7c9e95fc515e45" exitCode=0
Nov 02 12:44:30.777792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.777766    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" event={"ID":"89d69147-8068-4256-803f-5848a3d6b9f3","Type":"ContainerDied","Data":"e7e16e51471225ddd8dde93d4eca7ba348186340be8f5a1e8d7c9e95fc515e45"}
Nov 02 12:44:30.818543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.818514    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:44:30.976344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.976309    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\" (UniqueName: \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\") pod \"89d69147-8068-4256-803f-5848a3d6b9f3\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") "
Nov 02 12:44:30.976499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.976375    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\") pod \"89d69147-8068-4256-803f-5848a3d6b9f3\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") "
Nov 02 12:44:30.976713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.976687    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") pod \"89d69147-8068-4256-803f-5848a3d6b9f3\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") "
Nov 02 12:44:30.976785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.976732    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca" pod "89d69147-8068-4256-803f-5848a3d6b9f3" (UID: "89d69147-8068-4256-803f-5848a3d6b9f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:30.976785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.976750    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\") pod \"89d69147-8068-4256-803f-5848a3d6b9f3\" (UID: \"89d69147-8068-4256-803f-5848a3d6b9f3\") "
Nov 02 12:44:30.977492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.977452    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d69147-8068-4256-803f-5848a3d6b9f3-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:30.978704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.978676    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd" pod "89d69147-8068-4256-803f-5848a3d6b9f3" (UID: "89d69147-8068-4256-803f-5848a3d6b9f3"). InnerVolumeSpecName "kube-api-access-npbcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:30.978814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.978727    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs" pod "89d69147-8068-4256-803f-5848a3d6b9f3" (UID: "89d69147-8068-4256-803f-5848a3d6b9f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:30.978906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:30.978805    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls" pod "89d69147-8068-4256-803f-5848a3d6b9f3" (UID: "89d69147-8068-4256-803f-5848a3d6b9f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:31.079196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.079102    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:31.079411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.079392    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d69147-8068-4256-803f-5848a3d6b9f3-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:31.079505 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.079491    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbcd\" (UniqueName: \"kubernetes.io/projected/89d69147-8068-4256-803f-5848a3d6b9f3-kube-api-access-npbcd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:31.789370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.789339    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"
Nov 02 12:44:31.789929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.789382    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5" event={"ID":"89d69147-8068-4256-803f-5848a3d6b9f3","Type":"ContainerDied","Data":"79a1b2e4d00752ee77fdce6db3ee18bbeb5af4877d33c45adcd757932ac6f3aa"}
Nov 02 12:44:31.789929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.789442    4928 scope.go:122] "RemoveContainer" containerID="e7e16e51471225ddd8dde93d4eca7ba348186340be8f5a1e8d7c9e95fc515e45"
Nov 02 12:44:31.849245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.849216    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:44:31.864735 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:31.864690    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f7f01-5fd44479c6-gp7g5"]
Nov 02 12:44:32.509492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:32.509453    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d69147-8068-4256-803f-5848a3d6b9f3" path="/var/lib/kubelet/pods/89d69147-8068-4256-803f-5848a3d6b9f3/volumes"
Nov 02 12:44:35.152436 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.152368    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" podUID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" containerName="payload-pre-processing" containerID="cri-o://0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310" gracePeriod=25
Nov 02 12:44:35.453683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.451031    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:44:35.557643 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.557590    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\" (UniqueName: \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\") pod \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") "
Nov 02 12:44:35.557872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.557846    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\") pod \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\" (UID: \"267e6707-1fa7-4998-ba4f-b747adfd2ac4\") "
Nov 02 12:44:35.558363 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.558330    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume" pod "267e6707-1fa7-4998-ba4f-b747adfd2ac4" (UID: "267e6707-1fa7-4998-ba4f-b747adfd2ac4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:35.558750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.558727    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e6707-1fa7-4998-ba4f-b747adfd2ac4-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:35.560977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.560948    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8" pod "267e6707-1fa7-4998-ba4f-b747adfd2ac4" (UID: "267e6707-1fa7-4998-ba4f-b747adfd2ac4"). InnerVolumeSpecName "kube-api-access-qpjc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:35.660507 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.660462    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjc8\" (UniqueName: \"kubernetes.io/projected/267e6707-1fa7-4998-ba4f-b747adfd2ac4-kube-api-access-qpjc8\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:35.853117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.853008    4928 generic.go:356] "Generic (PLEG): container finished" podID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" containerID="0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310" exitCode=0
Nov 02 12:44:35.853117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.853071    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"
Nov 02 12:44:35.853117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.853100    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" event={"ID":"267e6707-1fa7-4998-ba4f-b747adfd2ac4","Type":"ContainerDied","Data":"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"}
Nov 02 12:44:35.853393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.853170    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7" event={"ID":"267e6707-1fa7-4998-ba4f-b747adfd2ac4","Type":"ContainerDied","Data":"f804393ac842127bd1320aef3f5755edc1315c99c5c7e250d914109c3ceac21b"}
Nov 02 12:44:35.853393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.853201    4928 scope.go:122] "RemoveContainer" containerID="0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"
Nov 02 12:44:35.867174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.867151    4928 scope.go:122] "RemoveContainer" containerID="0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"
Nov 02 12:44:35.867519 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:35.867496    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310\": container with ID starting with 0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310 not found: ID does not exist" containerID="0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"
Nov 02 12:44:35.867610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.867529    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310"} err="failed to get container status \"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310\": rpc error: code = NotFound desc = could not find container \"0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310\": container with ID starting with 0d7e650b36f69676aba513cff07b488a484fd8cdcc4578da9113853e249b7310 not found: ID does not exist"
Nov 02 12:44:35.917862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.916717    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:44:35.938719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:35.928541    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-97e08c-85f68cc5d7-6hzj7"]
Nov 02 12:44:36.509356 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:36.509320    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" path="/var/lib/kubelet/pods/267e6707-1fa7-4998-ba4f-b747adfd2ac4/volumes"
Nov 02 12:44:38.165846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.161237    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:44:38.165846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.161492    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" podUID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" containerName="maas-api" containerID="cri-o://f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650" gracePeriod=30
Nov 02 12:44:38.537109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.537051    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:44:38.654990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.650235    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\") pod \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") "
Nov 02 12:44:38.654990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.650475    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\") pod \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") "
Nov 02 12:44:38.654990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.650663    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\" (UniqueName: \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\") pod \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") "
Nov 02 12:44:38.654990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.650745    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\") pod \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\" (UID: \"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71\") "
Nov 02 12:44:38.654990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.652875    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca" pod "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" (UID: "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:38.675939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.675889    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd" pod "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" (UID: "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71"). InnerVolumeSpecName "kube-api-access-dcxwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:38.676095 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.676067    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs" pod "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" (UID: "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:38.677698 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.677671    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls" pod "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" (UID: "746c6062-3cdf-4f3d-9dd0-b0fdc3929c71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:38.753750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.753708    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:38.753750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.753750    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:38.753976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.753766    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxwd\" (UniqueName: \"kubernetes.io/projected/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-kube-api-access-dcxwd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:38.753976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.753781    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:38.884168 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.884131    4928 generic.go:356] "Generic (PLEG): container finished" podID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" containerID="f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650" exitCode=0
Nov 02 12:44:38.884336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.884177    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"
Nov 02 12:44:38.884336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.884183    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" event={"ID":"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71","Type":"ContainerDied","Data":"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"}
Nov 02 12:44:38.884336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.884237    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf" event={"ID":"746c6062-3cdf-4f3d-9dd0-b0fdc3929c71","Type":"ContainerDied","Data":"dd304ce51e140aed3df2d49ca72369aec9562b3143f3b929ea841bf6360d07e1"}
Nov 02 12:44:38.884336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.884269    4928 scope.go:122] "RemoveContainer" containerID="f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"
Nov 02 12:44:38.896093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.896073    4928 scope.go:122] "RemoveContainer" containerID="f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"
Nov 02 12:44:38.896350 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:38.896331    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650\": container with ID starting with f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650 not found: ID does not exist" containerID="f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"
Nov 02 12:44:38.896426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.896359    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650"} err="failed to get container status \"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650\": rpc error: code = NotFound desc = could not find container \"f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650\": container with ID starting with f5562e6cf822f44167967603eebb72560284295fb1b6d62248b965441c2ac650 not found: ID does not exist"
Nov 02 12:44:38.948731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.948697    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:44:38.958553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.958519    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97e08c-6b56f74d97-wztgf"]
Nov 02 12:44:38.993903 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.993868    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:44:38.995029 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:38.994142    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy" containerID="cri-o://5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4" gracePeriod=30
Nov 02 12:44:39.731053 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:39.731010    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:44:40.499175 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:40.499143    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" path="/var/lib/kubelet/pods/746c6062-3cdf-4f3d-9dd0-b0fdc3929c71/volumes"
Nov 02 12:44:44.313275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.313240    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:44:44.466140 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466100    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466198    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466268    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466324    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466375    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466452    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.466568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466461    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466568 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466557    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.466775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.466672    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.467116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.467081    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\") pod \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\" (UID: \"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6\") "
Nov 02 12:44:44.467939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.467908    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.468202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.468178    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.468432 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.468414    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.468732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.468706    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:44.470070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.470017    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.472519 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.472481    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "kube-api-access-kbt9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.475164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.475120    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.475308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.475154    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.475308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.475165    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo" pod "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" (UID: "36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:44:44.569777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569738    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569773    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569788    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569802    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569814    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569839    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569854    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.569977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.569866    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt9v\" (UniqueName: \"kubernetes.io/projected/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6-kube-api-access-kbt9v\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.991543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.991508    4928 generic.go:356] "Generic (PLEG): container finished" podID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerID="5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4" exitCode=0
Nov 02 12:44:44.991742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.991592    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" event={"ID":"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6","Type":"ContainerDied","Data":"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"}
Nov 02 12:44:44.991742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.991644    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m" event={"ID":"36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6","Type":"ContainerDied","Data":"ac49a43b37a3931415f5a7207cf267b096e94bd0ef815b8d1ab0331359720973"}
Nov 02 12:44:44.991742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.991673    4928 scope.go:122] "RemoveContainer" containerID="5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"
Nov 02 12:44:44.992561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:44.992029    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"
Nov 02 12:44:45.005861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.005834    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:44:45.006115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.006080    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" containerID="cri-o://fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171" gracePeriod=30
Nov 02 12:44:45.014338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.014322    4928 scope.go:122] "RemoveContainer" containerID="5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"
Nov 02 12:44:45.014648 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:45.014623    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4\": container with ID starting with 5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4 not found: ID does not exist" containerID="5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"
Nov 02 12:44:45.014740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.014655    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4"} err="failed to get container status \"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4\": rpc error: code = NotFound desc = could not find container \"5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4\": container with ID starting with 5e0b1e48b9e917cf1d90251306308f74d50bd9046f9dfbfc38b2b706fcccefd4 not found: ID does not exist"
Nov 02 12:44:45.097508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.097475    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:44:45.115856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:45.113230    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-28q8m"]
Nov 02 12:44:46.011640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:46.011604    4928 patch_prober.go:29] interesting pod/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:44:46.012095 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:46.011666    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:44:46.509960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:46.506201    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" path="/var/lib/kubelet/pods/36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6/volumes"
Nov 02 12:44:50.093943 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:50.093904    4928 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba55cd_ceaf_475a_a1da_5e300c3da861.slice/crio-conmon-fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:44:50.313963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.313930    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:44:50.369438 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369401    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369458    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369488    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369553    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369590    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369627    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369674    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369757    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369817    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\") pod \"52ba55cd-ceaf-475a-a1da-5e300c3da861\" (UID: \"52ba55cd-ceaf-475a-a1da-5e300c3da861\") "
Nov 02 12:44:50.369876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.369861    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:50.370154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370009    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:50.370154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370089    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:50.370271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370227    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:50.370326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370276    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:50.370565 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370543    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.370640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370566    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.370640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370576    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba55cd-ceaf-475a-a1da-5e300c3da861-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.370640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370586    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.370640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.370596    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.371782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.371745    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "kube-api-access-sh6lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:50.371976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.371952    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:50.372281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.372256    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:50.372390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.372369    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo" pod "52ba55cd-ceaf-475a-a1da-5e300c3da861" (UID: "52ba55cd-ceaf-475a-a1da-5e300c3da861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:44:50.471480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.471448    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6lt\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-kube-api-access-sh6lt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.471480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.471479    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.471480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.471489    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.471712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:50.471498    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba55cd-ceaf-475a-a1da-5e300c3da861-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:51.048660 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.048621    4928 generic.go:356] "Generic (PLEG): container finished" podID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerID="fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171" exitCode=0
Nov 02 12:44:51.048882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.048682    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"
Nov 02 12:44:51.048882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.048680    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerDied","Data":"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"}
Nov 02 12:44:51.048882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.048741    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554" event={"ID":"52ba55cd-ceaf-475a-a1da-5e300c3da861","Type":"ContainerDied","Data":"fa91e07a4c9f289c02054879941b9b61550a5578c3233b85c2bd477f00fe7030"}
Nov 02 12:44:51.048882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.048765    4928 scope.go:122] "RemoveContainer" containerID="fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"
Nov 02 12:44:51.060275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.060252    4928 scope.go:122] "RemoveContainer" containerID="eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"
Nov 02 12:44:51.070048 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.070016    4928 scope.go:122] "RemoveContainer" containerID="fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"
Nov 02 12:44:51.070306 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:51.070285    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171\": container with ID starting with fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171 not found: ID does not exist" containerID="fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"
Nov 02 12:44:51.070373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.070320    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171"} err="failed to get container status \"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171\": rpc error: code = NotFound desc = could not find container \"fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171\": container with ID starting with fd9ae55f71b032cf61158a2ad84ceddcc7f52ca3dabfa89b79730cc35ad1c171 not found: ID does not exist"
Nov 02 12:44:51.070373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.070344    4928 scope.go:122] "RemoveContainer" containerID="eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"
Nov 02 12:44:51.070595 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:51.070580    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab\": container with ID starting with eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab not found: ID does not exist" containerID="eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"
Nov 02 12:44:51.070665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.070597    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab"} err="failed to get container status \"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab\": rpc error: code = NotFound desc = could not find container \"eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab\": container with ID starting with eb1b4cb6235ce0bd2b0965e13677a254d4049aa20f15011ba403db22f38eb9ab not found: ID does not exist"
Nov 02 12:44:51.122400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.122371    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:44:51.139439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.139408    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-97e08c-openshift-default-f7c78f497-p9554"]
Nov 02 12:44:51.301692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.301599    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:44:51.302396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302374    4928 cpu_manager.go:414] "containerMap: removing container" podUID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" containerName="maas-api"
Nov 02 12:44:51.302396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302395    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" containerName="maas-api"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302417    4928 cpu_manager.go:414] "containerMap: removing container" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302425    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302437    4928 cpu_manager.go:414] "containerMap: removing container" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302445    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302458    4928 cpu_manager.go:414] "containerMap: removing container" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302464    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302479    4928 cpu_manager.go:414] "containerMap: removing container" podUID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" containerName="payload-pre-processing"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302487    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" containerName="payload-pre-processing"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302509    4928 cpu_manager.go:414] "containerMap: removing container" podUID="fb95eff3-e240-4acd-8eb2-3249dad451d4" containerName="maas-api"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302515    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb95eff3-e240-4acd-8eb2-3249dad451d4" containerName="maas-api"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302533    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302541    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302550    4928 cpu_manager.go:414] "containerMap: removing container" podUID="89d69147-8068-4256-803f-5848a3d6b9f3" containerName="maas-api"
Nov 02 12:44:51.302587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302557    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d69147-8068-4256-803f-5848a3d6b9f3" containerName="maas-api"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302729    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="746c6062-3cdf-4f3d-9dd0-b0fdc3929c71" containerName="maas-api"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302746    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302756    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b0b05f-a3e3-48bf-92dd-e7e6fa0f4ae6" containerName="istio-proxy"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302764    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" containerName="istio-proxy"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302774    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="267e6707-1fa7-4998-ba4f-b747adfd2ac4" containerName="payload-pre-processing"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302786    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb95eff3-e240-4acd-8eb2-3249dad451d4" containerName="maas-api"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302797    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3125fe2e-a444-4f92-a12e-4e02fac81321" containerName="payload-processing"
Nov 02 12:44:51.303245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.302806    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d69147-8068-4256-803f-5848a3d6b9f3" containerName="maas-api"
Nov 02 12:44:51.308327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.308306    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.324351 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.324326    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0cf7e-serving-cert\""
Nov 02 12:44:51.331491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.331465    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:44:51.380803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.380762    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgl5\" (UniqueName: \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.381004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.380861    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.381004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.380907    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.381076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.381036    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.482204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.482168    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgl5\" (UniqueName: \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.482383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.482227    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.482383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.482266    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.482383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.482371    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.483107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.483075    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.484803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.484772    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.484965 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.484881    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.506727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.506698    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgl5\" (UniqueName: \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\") pod \"maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.630491 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.630396    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:51.983798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:51.983764    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:44:51.984458 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:44:51.984426    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7983f6ed_cfd2_43e1_85ec_284d6c6cd37e.slice/crio-fe323a64eee073dc56844a7c023d34d9ad94453926ead96f46d3b8c3749c20dc WatchSource:0}: Error finding container fe323a64eee073dc56844a7c023d34d9ad94453926ead96f46d3b8c3749c20dc: Status 404 returned error can't find the container with id fe323a64eee073dc56844a7c023d34d9ad94453926ead96f46d3b8c3749c20dc
Nov 02 12:44:52.065254 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:52.065196    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" event={"ID":"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e","Type":"ContainerStarted","Data":"fe323a64eee073dc56844a7c023d34d9ad94453926ead96f46d3b8c3749c20dc"}
Nov 02 12:44:52.499930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:52.499893    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba55cd-ceaf-475a-a1da-5e300c3da861" path="/var/lib/kubelet/pods/52ba55cd-ceaf-475a-a1da-5e300c3da861/volumes"
Nov 02 12:44:52.519318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:52.519281    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:44:52.593556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:52.593515    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:44:53.083021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:53.082977    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" event={"ID":"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e","Type":"ContainerStarted","Data":"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"}
Nov 02 12:44:53.083266 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:53.083122    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:53.136181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:53.131857    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" podStartSLOduration=1.752773261 podStartE2EDuration="2.131821693s" totalImagesPullingTime="379.048432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:53.126602499 +0000 UTC m=+1090.775656472" watchObservedRunningTime="2026-11-02 12:44:53.131821693 +0000 UTC m=+1090.780875654"
Nov 02 12:44:57.519620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.519560    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" podUID="b5271259-bfa2-4f75-8fab-cf28eca3455b" containerName="payload-processing" containerID="cri-o://e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422" gracePeriod=25
Nov 02 12:44:57.594324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.594276    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerName="payload-pre-processing" containerID="cri-o://a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218" gracePeriod=25
Nov 02 12:44:57.817976 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:44:57.817912    4928 logging.go:55] [core] [Channel #980 SubChannel #981]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.114:50052", ServerName: "10.132.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.114:50052: connect: connection refused"
Nov 02 12:44:57.911477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.911443    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:44:57.938923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.938899    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:44:57.947137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.947092    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:44:57.947395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:57.947342    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" podUID="74f84930-febf-40ed-a33b-6b0120b71e4c" containerName="maas-api" containerID="cri-o://d683f04fe6448be5a787e0b3d9c9161ff3434c28657140c3dcb17dc02da062b4" gracePeriod=30
Nov 02 12:44:58.013128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.013055    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\" (UniqueName: \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\") pod \"2082d000-5279-4b2b-bc0a-d4e23a9da359\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") "
Nov 02 12:44:58.013312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.013147    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\") pod \"2082d000-5279-4b2b-bc0a-d4e23a9da359\" (UID: \"2082d000-5279-4b2b-bc0a-d4e23a9da359\") "
Nov 02 12:44:58.013312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.013227    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\") pod \"b5271259-bfa2-4f75-8fab-cf28eca3455b\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") "
Nov 02 12:44:58.013312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.013280    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\" (UniqueName: \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\") pod \"b5271259-bfa2-4f75-8fab-cf28eca3455b\" (UID: \"b5271259-bfa2-4f75-8fab-cf28eca3455b\") "
Nov 02 12:44:58.015023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.014696    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume" pod "2082d000-5279-4b2b-bc0a-d4e23a9da359" (UID: "2082d000-5279-4b2b-bc0a-d4e23a9da359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:58.015023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.014985    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume" pod "b5271259-bfa2-4f75-8fab-cf28eca3455b" (UID: "b5271259-bfa2-4f75-8fab-cf28eca3455b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:58.015490 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.015467    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2082d000-5279-4b2b-bc0a-d4e23a9da359-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.015580 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.015493    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5271259-bfa2-4f75-8fab-cf28eca3455b-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.016626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.016597    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst" pod "b5271259-bfa2-4f75-8fab-cf28eca3455b" (UID: "b5271259-bfa2-4f75-8fab-cf28eca3455b"). InnerVolumeSpecName "kube-api-access-ntfst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:58.021962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.021933    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc" pod "2082d000-5279-4b2b-bc0a-d4e23a9da359" (UID: "2082d000-5279-4b2b-bc0a-d4e23a9da359"). InnerVolumeSpecName "kube-api-access-wqjvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:58.116990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.116888    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfst\" (UniqueName: \"kubernetes.io/projected/b5271259-bfa2-4f75-8fab-cf28eca3455b-kube-api-access-ntfst\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.116990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.116934    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqjvc\" (UniqueName: \"kubernetes.io/projected/2082d000-5279-4b2b-bc0a-d4e23a9da359-kube-api-access-wqjvc\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.121878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.121819    4928 generic.go:356] "Generic (PLEG): container finished" podID="b5271259-bfa2-4f75-8fab-cf28eca3455b" containerID="e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422" exitCode=0
Nov 02 12:44:58.121878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.121859    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" event={"ID":"b5271259-bfa2-4f75-8fab-cf28eca3455b","Type":"ContainerDied","Data":"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"}
Nov 02 12:44:58.122021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.121897    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"
Nov 02 12:44:58.122021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.121925    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7" event={"ID":"b5271259-bfa2-4f75-8fab-cf28eca3455b","Type":"ContainerDied","Data":"41b37191072155079c57f016ec887369a3c49f4470daed83d85d16228d6aea8d"}
Nov 02 12:44:58.122021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.121950    4928 scope.go:122] "RemoveContainer" containerID="e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"
Nov 02 12:44:58.123715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.123694    4928 generic.go:356] "Generic (PLEG): container finished" podID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerID="a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218" exitCode=0
Nov 02 12:44:58.123963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.123745    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"
Nov 02 12:44:58.123963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.123820    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" event={"ID":"2082d000-5279-4b2b-bc0a-d4e23a9da359","Type":"ContainerDied","Data":"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"}
Nov 02 12:44:58.123963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.123899    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" event={"ID":"2082d000-5279-4b2b-bc0a-d4e23a9da359","Type":"ContainerDied","Data":"35e1363fdaed7286f5de144dd34853190d4f4552008f5c7f11e3abcee6c3330b"}
Nov 02 12:44:58.126054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.126030    4928 generic.go:356] "Generic (PLEG): container finished" podID="74f84930-febf-40ed-a33b-6b0120b71e4c" containerID="d683f04fe6448be5a787e0b3d9c9161ff3434c28657140c3dcb17dc02da062b4" exitCode=0
Nov 02 12:44:58.126144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.126129    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" event={"ID":"74f84930-febf-40ed-a33b-6b0120b71e4c","Type":"ContainerDied","Data":"d683f04fe6448be5a787e0b3d9c9161ff3434c28657140c3dcb17dc02da062b4"}
Nov 02 12:44:58.141783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.141760    4928 scope.go:122] "RemoveContainer" containerID="e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"
Nov 02 12:44:58.143295 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:58.143148    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422\": container with ID starting with e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422 not found: ID does not exist" containerID="e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"
Nov 02 12:44:58.143295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.143184    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422"} err="failed to get container status \"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422\": rpc error: code = NotFound desc = could not find container \"e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422\": container with ID starting with e461be93e43ae7cb932eea87d14214b0c25a16518b865411087e827c914dd422 not found: ID does not exist"
Nov 02 12:44:58.143295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.143206    4928 scope.go:122] "RemoveContainer" containerID="a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"
Nov 02 12:44:58.162483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.162446    4928 scope.go:122] "RemoveContainer" containerID="a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"
Nov 02 12:44:58.163073 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:44:58.163050    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218\": container with ID starting with a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218 not found: ID does not exist" containerID="a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"
Nov 02 12:44:58.163227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.163202    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218"} err="failed to get container status \"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218\": rpc error: code = NotFound desc = could not find container \"a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218\": container with ID starting with a15b38007cc16577f5b7318cf189decbba11feb8a51a7503c9c085b5fcd90218 not found: ID does not exist"
Nov 02 12:44:58.209876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.207777    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:44:58.214246 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.214184    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj"]
Nov 02 12:44:58.251893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.248893    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:44:58.251893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.251759    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a3af1-5f4d9f56f-7ggm7"]
Nov 02 12:44:58.271224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.271150    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:44:58.420973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.420853    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\") pod \"74f84930-febf-40ed-a33b-6b0120b71e4c\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") "
Nov 02 12:44:58.421154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.421021    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\") pod \"74f84930-febf-40ed-a33b-6b0120b71e4c\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") "
Nov 02 12:44:58.421440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.421408    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca" pod "74f84930-febf-40ed-a33b-6b0120b71e4c" (UID: "74f84930-febf-40ed-a33b-6b0120b71e4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:58.421506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.421481    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\" (UniqueName: \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\") pod \"74f84930-febf-40ed-a33b-6b0120b71e4c\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") "
Nov 02 12:44:58.421550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.421526    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\") pod \"74f84930-febf-40ed-a33b-6b0120b71e4c\" (UID: \"74f84930-febf-40ed-a33b-6b0120b71e4c\") "
Nov 02 12:44:58.422767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.422741    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f84930-febf-40ed-a33b-6b0120b71e4c-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.424468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.424440    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls" pod "74f84930-febf-40ed-a33b-6b0120b71e4c" (UID: "74f84930-febf-40ed-a33b-6b0120b71e4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:58.424728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.424700    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs" pod "74f84930-febf-40ed-a33b-6b0120b71e4c" (UID: "74f84930-febf-40ed-a33b-6b0120b71e4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:58.433798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.424935    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b" pod "74f84930-febf-40ed-a33b-6b0120b71e4c" (UID: "74f84930-febf-40ed-a33b-6b0120b71e4c"). InnerVolumeSpecName "kube-api-access-th24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:58.501117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.501076    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" path="/var/lib/kubelet/pods/2082d000-5279-4b2b-bc0a-d4e23a9da359/volumes"
Nov 02 12:44:58.501694 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.501672    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5271259-bfa2-4f75-8fab-cf28eca3455b" path="/var/lib/kubelet/pods/b5271259-bfa2-4f75-8fab-cf28eca3455b/volumes"
Nov 02 12:44:58.524188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.524151    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.524188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.524183    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th24b\" (UniqueName: \"kubernetes.io/projected/74f84930-febf-40ed-a33b-6b0120b71e4c-kube-api-access-th24b\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.524188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.524194    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f84930-febf-40ed-a33b-6b0120b71e4c-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:44:58.818561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:58.818509    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a3af1-68cdf5d96c-27tqj" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.114:50052\" within 1s: context deadline exceeded"
Nov 02 12:44:58.818735 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:44:58.818642    4928 logging.go:55] [core] [Channel #980 SubChannel #981]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.114:50052", ServerName: "10.132.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.114:50052: operation was canceled"
Nov 02 12:44:59.095993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.095917    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:44:59.138536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.138483    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks" event={"ID":"74f84930-febf-40ed-a33b-6b0120b71e4c","Type":"ContainerDied","Data":"c6555f7b305c5f622d30bbc79c7cf76775c3bfb42b7dd399396df7044f038d95"}
Nov 02 12:44:59.138536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.138519    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"
Nov 02 12:44:59.138780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.138569    4928 scope.go:122] "RemoveContainer" containerID="d683f04fe6448be5a787e0b3d9c9161ff3434c28657140c3dcb17dc02da062b4"
Nov 02 12:44:59.201962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.201927    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:44:59.207093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:44:59.207065    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c79cde-58d456b7-5jjks"]
Nov 02 12:45:00.180215 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180180    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:00.180857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180837    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b5271259-bfa2-4f75-8fab-cf28eca3455b" containerName="payload-processing"
Nov 02 12:45:00.180917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180858    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5271259-bfa2-4f75-8fab-cf28eca3455b" containerName="payload-processing"
Nov 02 12:45:00.180917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180879    4928 cpu_manager.go:414] "containerMap: removing container" podUID="74f84930-febf-40ed-a33b-6b0120b71e4c" containerName="maas-api"
Nov 02 12:45:00.180917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180885    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f84930-febf-40ed-a33b-6b0120b71e4c" containerName="maas-api"
Nov 02 12:45:00.180917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180905    4928 cpu_manager.go:414] "containerMap: removing container" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerName="payload-pre-processing"
Nov 02 12:45:00.180917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.180911    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerName="payload-pre-processing"
Nov 02 12:45:00.181071 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.181060    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5271259-bfa2-4f75-8fab-cf28eca3455b" containerName="payload-processing"
Nov 02 12:45:00.181146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.181073    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f84930-febf-40ed-a33b-6b0120b71e4c" containerName="maas-api"
Nov 02 12:45:00.181146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.181083    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="2082d000-5279-4b2b-bc0a-d4e23a9da359" containerName="payload-pre-processing"
Nov 02 12:45:00.186762 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.186732    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.222022 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.221987    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:00.232515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.229672    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:00.232515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.229788    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.248192 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.240248    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:00.250120 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.249968    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.261987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.261051    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.261987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.261093    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7kx\" (UniqueName: \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.265956 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.265931    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:00.273419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.273354    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:00.294877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.294276    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:45:00.363216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363182    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.363216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363221    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.363538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.363538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363351    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7kx\" (UniqueName: \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.363538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363396    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmc2\" (UniqueName: \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.363538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.363524    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ccs\" (UniqueName: \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.364186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.364165    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.388953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.388914    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7kx\" (UniqueName: \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.465542 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.465456    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.465542 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.465497    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.465742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.465568    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmc2\" (UniqueName: \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.465742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.465719    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ccs\" (UniqueName: \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.466375 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.466352    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.466431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.466352    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.491427 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.491399    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmc2\" (UniqueName: \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.491555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.491488    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ccs\" (UniqueName: \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.499409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.499383    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f84930-febf-40ed-a33b-6b0120b71e4c" path="/var/lib/kubelet/pods/74f84930-febf-40ed-a33b-6b0120b71e4c/volumes"
Nov 02 12:45:00.507925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.507903    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:00.566156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.566047    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:00.583002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.582973    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:00.837462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.837417    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:00.850273 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:45:00.850236    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701ea9e0_a746_4cbf_878d_ce964d40adbe.slice/crio-4aabe35351c466f5a7c49a9df67967af335c0d659717e2180bc782dc4ff10d67 WatchSource:0}: Error finding container 4aabe35351c466f5a7c49a9df67967af335c0d659717e2180bc782dc4ff10d67: Status 404 returned error can't find the container with id 4aabe35351c466f5a7c49a9df67967af335c0d659717e2180bc782dc4ff10d67
Nov 02 12:45:00.901982 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.901954    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:00.902520 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:45:00.902493    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798df37d_dd79_4c18_8b33_f3479a609f9b.slice/crio-9f5805fc4b86f3ec7420474862fc5bf33c8bbbd96e7caa4376dbddc75dea38e7 WatchSource:0}: Error finding container 9f5805fc4b86f3ec7420474862fc5bf33c8bbbd96e7caa4376dbddc75dea38e7: Status 404 returned error can't find the container with id 9f5805fc4b86f3ec7420474862fc5bf33c8bbbd96e7caa4376dbddc75dea38e7
Nov 02 12:45:00.924760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:00.924730    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:01.038564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:01.038500    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"]
Nov 02 12:45:01.044439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:01.044399    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-993035a6-vfk4g"]
Nov 02 12:45:01.168534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:01.168493    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerStarted","Data":"b490315f83497ea210a5bdc4fe5a16b475d225bdb310661b4db3dca64afb5303"}
Nov 02 12:45:01.171347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:01.171312    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerStarted","Data":"4aabe35351c466f5a7c49a9df67967af335c0d659717e2180bc782dc4ff10d67"}
Nov 02 12:45:01.173416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:01.173390    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerStarted","Data":"9f5805fc4b86f3ec7420474862fc5bf33c8bbbd96e7caa4376dbddc75dea38e7"}
Nov 02 12:45:02.183926 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.183820    4928 generic.go:356] "Generic (PLEG): container finished" podID="23e58130-05cf-4096-8f92-e6248eb87722" containerID="cfc9951a9a5fda0897e172dd74d4e2ba3be8e956be57161cf539f6fcfd180511" exitCode=60
Nov 02 12:45:02.184461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.183924    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerDied","Data":"cfc9951a9a5fda0897e172dd74d4e2ba3be8e956be57161cf539f6fcfd180511"}
Nov 02 12:45:02.184461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.184271    4928 scope.go:122] "RemoveContainer" containerID="cfc9951a9a5fda0897e172dd74d4e2ba3be8e956be57161cf539f6fcfd180511"
Nov 02 12:45:02.186524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.186497    4928 generic.go:356] "Generic (PLEG): container finished" podID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerID="b426122631a94ba5971dda9e88107fb2983d830a7c506d4c79b73441259bdf7e" exitCode=60
Nov 02 12:45:02.186650 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.186533    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerDied","Data":"b426122631a94ba5971dda9e88107fb2983d830a7c506d4c79b73441259bdf7e"}
Nov 02 12:45:02.186918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.186894    4928 scope.go:122] "RemoveContainer" containerID="b426122631a94ba5971dda9e88107fb2983d830a7c506d4c79b73441259bdf7e"
Nov 02 12:45:02.188535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.188497    4928 generic.go:356] "Generic (PLEG): container finished" podID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerID="d47c3de992827217e0a04983c4d7df77f21bea4c0230fe2b950829bae79d3083" exitCode=60
Nov 02 12:45:02.188535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.188524    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerDied","Data":"d47c3de992827217e0a04983c4d7df77f21bea4c0230fe2b950829bae79d3083"}
Nov 02 12:45:02.188950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.188924    4928 scope.go:122] "RemoveContainer" containerID="d47c3de992827217e0a04983c4d7df77f21bea4c0230fe2b950829bae79d3083"
Nov 02 12:45:02.509656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.509607    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79616ada-c240-438f-9a2d-190563551866" path="/var/lib/kubelet/pods/79616ada-c240-438f-9a2d-190563551866/volumes"
Nov 02 12:45:02.619552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.619518    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:45:02.626270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.626237    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.654221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.654119    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:45:02.661576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.661539    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e6ceda-openshift-default-dockercfg-mpb9j\""
Nov 02 12:45:02.814482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814437    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814584    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jhw\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814647    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814696    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814727    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814754    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814839    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.814880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814874    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.815207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.814918    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.918275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.918240    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.918538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.918507    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.918653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.918603    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.918886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.918865    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.919962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.919058    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jhw\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.919962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.919133    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.919962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.919197    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.919962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.919241    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.919962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.919282    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.920312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.920157    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.921153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.921121    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.921704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.921649    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.922495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.922309    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.922586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.922544    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.931128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.931104    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.931971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.931951    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.943531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.943511    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:02.945256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:02.945234    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jhw\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\") pod \"e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") " pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:03.204912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.204871    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerStarted","Data":"dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"}
Nov 02 12:45:03.207287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.207252    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerStarted","Data":"6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"}
Nov 02 12:45:03.210013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.209974    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerStarted","Data":"2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"}
Nov 02 12:45:03.245404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.245373    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:03.248216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.248166    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" podStartSLOduration=3.248152604 podStartE2EDuration="3.248152604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:03.246442599 +0000 UTC m=+1100.895496557" watchObservedRunningTime="2026-11-02 12:45:03.248152604 +0000 UTC m=+1100.897206562"
Nov 02 12:45:03.288556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.288500    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" podStartSLOduration=3.288485808 podStartE2EDuration="3.288485808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:03.287646462 +0000 UTC m=+1100.936700420" watchObservedRunningTime="2026-11-02 12:45:03.288485808 +0000 UTC m=+1100.937539796"
Nov 02 12:45:03.335272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.335074    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" podStartSLOduration=3.335056544 podStartE2EDuration="3.335056544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:03.331940115 +0000 UTC m=+1100.980994076" watchObservedRunningTime="2026-11-02 12:45:03.335056544 +0000 UTC m=+1100.984110503"
Nov 02 12:45:03.604224 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:45:03.604188    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1bbde6_b3a4_4403_bd15_f0e07ab10543.slice/crio-6f33484bf1d25c9f8898b2227d0951b1e1472a6964a7366269f4a4e7c6895e2f WatchSource:0}: Error finding container 6f33484bf1d25c9f8898b2227d0951b1e1472a6964a7366269f4a4e7c6895e2f: Status 404 returned error can't find the container with id 6f33484bf1d25c9f8898b2227d0951b1e1472a6964a7366269f4a4e7c6895e2f
Nov 02 12:45:03.604354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.604224    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:45:03.606936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.606897    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:03.607072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.606984    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:03.607072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:03.607031    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:04.220122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.220072    4928 generic.go:356] "Generic (PLEG): container finished" podID="23e58130-05cf-4096-8f92-e6248eb87722" containerID="2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f" exitCode=60
Nov 02 12:45:04.220625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.220155    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerDied","Data":"2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"}
Nov 02 12:45:04.220625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.220275    4928 scope.go:122] "RemoveContainer" containerID="cfc9951a9a5fda0897e172dd74d4e2ba3be8e956be57161cf539f6fcfd180511"
Nov 02 12:45:04.220952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.220925    4928 scope.go:122] "RemoveContainer" containerID="2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"
Nov 02 12:45:04.221463 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:04.221348    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t_odh-ai-gateway-infra(23e58130-05cf-4096-8f92-e6248eb87722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" podUID="23e58130-05cf-4096-8f92-e6248eb87722"
Nov 02 12:45:04.222586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.222557    4928 generic.go:356] "Generic (PLEG): container finished" podID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerID="dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e" exitCode=60
Nov 02 12:45:04.222689 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.222624    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerDied","Data":"dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"}
Nov 02 12:45:04.222979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.222953    4928 scope.go:122] "RemoveContainer" containerID="dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"
Nov 02 12:45:04.223318 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:04.223290    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk_odh-ai-gateway-infra(701ea9e0-a746-4cbf-878d-ce964d40adbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe"
Nov 02 12:45:04.225082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.225049    4928 generic.go:356] "Generic (PLEG): container finished" podID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerID="6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22" exitCode=60
Nov 02 12:45:04.225182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.225108    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerDied","Data":"6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"}
Nov 02 12:45:04.225408 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.225391    4928 scope.go:122] "RemoveContainer" containerID="6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"
Nov 02 12:45:04.225729 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:04.225706    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm_odh-ai-gateway-infra(798df37d-dd79-4c18-8b33-f3479a609f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b"
Nov 02 12:45:04.227135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.227102    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerStarted","Data":"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"}
Nov 02 12:45:04.227233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.227160    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerStarted","Data":"6f33484bf1d25c9f8898b2227d0951b1e1472a6964a7366269f4a4e7c6895e2f"}
Nov 02 12:45:04.236135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.236116    4928 scope.go:122] "RemoveContainer" containerID="b426122631a94ba5971dda9e88107fb2983d830a7c506d4c79b73441259bdf7e"
Nov 02 12:45:04.246489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.246471    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:04.250223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.249897    4928 scope.go:122] "RemoveContainer" containerID="d47c3de992827217e0a04983c4d7df77f21bea4c0230fe2b950829bae79d3083"
Nov 02 12:45:04.253105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.253087    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:04.312227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:04.312173    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podStartSLOduration=2.312157528 podStartE2EDuration="2.312157528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:04.309073853 +0000 UTC m=+1101.958127812" watchObservedRunningTime="2026-11-02 12:45:04.312157528 +0000 UTC m=+1101.961211483"
Nov 02 12:45:05.239182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:05.239150    4928 scope.go:122] "RemoveContainer" containerID="2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"
Nov 02 12:45:05.239644 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:05.239558    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t_odh-ai-gateway-infra(23e58130-05cf-4096-8f92-e6248eb87722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" podUID="23e58130-05cf-4096-8f92-e6248eb87722"
Nov 02 12:45:05.241162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:05.241135    4928 scope.go:122] "RemoveContainer" containerID="dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"
Nov 02 12:45:05.241503 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:05.241475    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk_odh-ai-gateway-infra(701ea9e0-a746-4cbf-878d-ce964d40adbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe"
Nov 02 12:45:05.242963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:05.242941    4928 scope.go:122] "RemoveContainer" containerID="6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"
Nov 02 12:45:05.243123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:05.243101    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:05.243259 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:05.243240    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm_odh-ai-gateway-infra(798df37d-dd79-4c18-8b33-f3479a609f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b"
Nov 02 12:45:05.244148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:05.244125    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:09.730974 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:09.730933    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:45:13.490464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:13.490431    4928 scope.go:122] "RemoveContainer" containerID="2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"
Nov 02 12:45:13.491372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:13.490571    4928 scope.go:122] "RemoveContainer" containerID="dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"
Nov 02 12:45:14.490620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.490586    4928 scope.go:122] "RemoveContainer" containerID="6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"
Nov 02 12:45:14.551729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.551691    4928 generic.go:356] "Generic (PLEG): container finished" podID="23e58130-05cf-4096-8f92-e6248eb87722" containerID="f9cb6f84698bd91872e9712493348ff19eeec157545f874386af6751e150ae4e" exitCode=60
Nov 02 12:45:14.551876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.551767    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerDied","Data":"f9cb6f84698bd91872e9712493348ff19eeec157545f874386af6751e150ae4e"}
Nov 02 12:45:14.551876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.551862    4928 scope.go:122] "RemoveContainer" containerID="2475744049c747906698098c18476f8cda61390221ce94fcbd19ffd0a4aed89f"
Nov 02 12:45:14.552390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.552368    4928 scope.go:122] "RemoveContainer" containerID="f9cb6f84698bd91872e9712493348ff19eeec157545f874386af6751e150ae4e"
Nov 02 12:45:14.552870 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:14.552804    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t_odh-ai-gateway-infra(23e58130-05cf-4096-8f92-e6248eb87722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" podUID="23e58130-05cf-4096-8f92-e6248eb87722"
Nov 02 12:45:14.554010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.553972    4928 generic.go:356] "Generic (PLEG): container finished" podID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerID="261bf0fe9b71e832a8b9ab87d6abe6174150dbfe9ba04ec86112cb28c74cb072" exitCode=60
Nov 02 12:45:14.554096 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.554035    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerDied","Data":"261bf0fe9b71e832a8b9ab87d6abe6174150dbfe9ba04ec86112cb28c74cb072"}
Nov 02 12:45:14.554421 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.554406    4928 scope.go:122] "RemoveContainer" containerID="261bf0fe9b71e832a8b9ab87d6abe6174150dbfe9ba04ec86112cb28c74cb072"
Nov 02 12:45:14.554733 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:14.554712    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk_odh-ai-gateway-infra(701ea9e0-a746-4cbf-878d-ce964d40adbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe"
Nov 02 12:45:14.673854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:14.673816    4928 scope.go:122] "RemoveContainer" containerID="dd97f484cd2a9cd389394794be6d8f1c63482203ec29c7b58084714ec196475e"
Nov 02 12:45:15.050583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.050540    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"]
Nov 02 12:45:15.055619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.055593    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faea81e9-9dhvv"]
Nov 02 12:45:15.568983 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.568947    4928 generic.go:356] "Generic (PLEG): container finished" podID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerID="f63391ca210ae2d880e2a147e0c05002094a0cdbf9cef95a61aeaca921276384" exitCode=60
Nov 02 12:45:15.569446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.568997    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerDied","Data":"f63391ca210ae2d880e2a147e0c05002094a0cdbf9cef95a61aeaca921276384"}
Nov 02 12:45:15.569446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.569068    4928 scope.go:122] "RemoveContainer" containerID="6d47e635c3a26161885a3f6cc0acc7e8855c104d9bc4c9f156e736e0a9240a22"
Nov 02 12:45:15.569553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.569448    4928 scope.go:122] "RemoveContainer" containerID="f63391ca210ae2d880e2a147e0c05002094a0cdbf9cef95a61aeaca921276384"
Nov 02 12:45:15.569874 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:15.569799    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm_odh-ai-gateway-infra(798df37d-dd79-4c18-8b33-f3479a609f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b"
Nov 02 12:45:15.608550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.608521    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:15.649193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.649157    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:15.803665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.803636    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:15.886037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.885912    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:15.900172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.899497    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\") pod \"23e58130-05cf-4096-8f92-e6248eb87722\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") "
Nov 02 12:45:15.900172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.899563    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\" (UniqueName: \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\") pod \"23e58130-05cf-4096-8f92-e6248eb87722\" (UID: \"23e58130-05cf-4096-8f92-e6248eb87722\") "
Nov 02 12:45:15.900172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.899906    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca" pod "23e58130-05cf-4096-8f92-e6248eb87722" (UID: "23e58130-05cf-4096-8f92-e6248eb87722"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:15.900600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.900556    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e58130-05cf-4096-8f92-e6248eb87722-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:15.903890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:15.902903    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs" pod "23e58130-05cf-4096-8f92-e6248eb87722" (UID: "23e58130-05cf-4096-8f92-e6248eb87722"). InnerVolumeSpecName "kube-api-access-q8ccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:16.001379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.001296    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\" (UniqueName: \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\") pod \"701ea9e0-a746-4cbf-878d-ce964d40adbe\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") "
Nov 02 12:45:16.001563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.001400    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\") pod \"701ea9e0-a746-4cbf-878d-ce964d40adbe\" (UID: \"701ea9e0-a746-4cbf-878d-ce964d40adbe\") "
Nov 02 12:45:16.001842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.001794    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca" pod "701ea9e0-a746-4cbf-878d-ce964d40adbe" (UID: "701ea9e0-a746-4cbf-878d-ce964d40adbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:16.002030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.002015    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701ea9e0-a746-4cbf-878d-ce964d40adbe-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.002070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.002034    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ccs\" (UniqueName: \"kubernetes.io/projected/23e58130-05cf-4096-8f92-e6248eb87722-kube-api-access-q8ccs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.022643 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.021775    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx" pod "701ea9e0-a746-4cbf-878d-ce964d40adbe" (UID: "701ea9e0-a746-4cbf-878d-ce964d40adbe"). InnerVolumeSpecName "kube-api-access-7d7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:16.049125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049093    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:45:16.049882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049864    4928 cpu_manager.go:414] "containerMap: removing container" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.049948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049883    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.049948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049898    4928 cpu_manager.go:414] "containerMap: removing container" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.049948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049924    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.049948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049948    4928 cpu_manager.go:414] "containerMap: removing container" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.050076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049954    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.050076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049966    4928 cpu_manager.go:414] "containerMap: removing container" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.050076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.049971    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.050154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.050100    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.050154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.050110    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.050154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.050118    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.050154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.050125    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:16.050154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.050135    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:16.054089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.054067    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.084380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.084352    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0cf7e-openshift-default-dockercfg-s488n\""
Nov 02 12:45:16.100516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.100481    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:45:16.103873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.103840    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7kx\" (UniqueName: \"kubernetes.io/projected/701ea9e0-a746-4cbf-878d-ce964d40adbe-kube-api-access-7d7kx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.204916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.204873    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.204929    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205048    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205106    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205187    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205237    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trpkt\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205367    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205432    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.205492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.205479    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.306642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306559    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.306642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306619    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpkt\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.306905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306711    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.306905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306778    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.306905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306815    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.306974    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307023    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307063    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307125    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307279 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307245    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307565    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307730    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307774    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.307805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.307788    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.309242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.309212    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.309535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.309516    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.334208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.334176    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trpkt\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.338575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.338551    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.368647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.368618    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:16.500336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.500295    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e229eda-d1ff-4fec-8c01-b3cac6221902" path="/var/lib/kubelet/pods/4e229eda-d1ff-4fec-8c01-b3cac6221902/volumes"
Nov 02 12:45:16.579379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.579306    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"
Nov 02 12:45:16.579379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.579323    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t" event={"ID":"23e58130-05cf-4096-8f92-e6248eb87722","Type":"ContainerDied","Data":"b490315f83497ea210a5bdc4fe5a16b475d225bdb310661b4db3dca64afb5303"}
Nov 02 12:45:16.579910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.579380    4928 scope.go:122] "RemoveContainer" containerID="f9cb6f84698bd91872e9712493348ff19eeec157545f874386af6751e150ae4e"
Nov 02 12:45:16.580986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.580959    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"
Nov 02 12:45:16.581105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.580994    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk" event={"ID":"701ea9e0-a746-4cbf-878d-ce964d40adbe","Type":"ContainerDied","Data":"4aabe35351c466f5a7c49a9df67967af335c0d659717e2180bc782dc4ff10d67"}
Nov 02 12:45:16.590638 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.590620    4928 scope.go:122] "RemoveContainer" containerID="261bf0fe9b71e832a8b9ab87d6abe6174150dbfe9ba04ec86112cb28c74cb072"
Nov 02 12:45:16.630126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.630088    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:16.652556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.652528    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:16.665183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.665155    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893725-z7rkk"]
Nov 02 12:45:16.693297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.693268    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:16.698545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.698519    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893725-2qp4t"]
Nov 02 12:45:16.724809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.723256    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:45:16.736639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.736598    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:16.736924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.736691    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:16.736924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.736740    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:16.870983 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:16.870959    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:17.021887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.019355    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\" (UniqueName: \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\") pod \"798df37d-dd79-4c18-8b33-f3479a609f9b\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") "
Nov 02 12:45:17.021887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.019537    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\") pod \"798df37d-dd79-4c18-8b33-f3479a609f9b\" (UID: \"798df37d-dd79-4c18-8b33-f3479a609f9b\") "
Nov 02 12:45:17.021887 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.020804    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca" pod "798df37d-dd79-4c18-8b33-f3479a609f9b" (UID: "798df37d-dd79-4c18-8b33-f3479a609f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:17.032053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.022795    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798df37d-dd79-4c18-8b33-f3479a609f9b-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:17.059850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.041406    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2" pod "798df37d-dd79-4c18-8b33-f3479a609f9b" (UID: "798df37d-dd79-4c18-8b33-f3479a609f9b"). InnerVolumeSpecName "kube-api-access-qwmc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:17.124233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.124081    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwmc2\" (UniqueName: \"kubernetes.io/projected/798df37d-dd79-4c18-8b33-f3479a609f9b-kube-api-access-qwmc2\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:45:17.595048 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.595016    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"
Nov 02 12:45:17.595557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.595013    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm" event={"ID":"798df37d-dd79-4c18-8b33-f3479a609f9b","Type":"ContainerDied","Data":"9f5805fc4b86f3ec7420474862fc5bf33c8bbbd96e7caa4376dbddc75dea38e7"}
Nov 02 12:45:17.595557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.595157    4928 scope.go:122] "RemoveContainer" containerID="f63391ca210ae2d880e2a147e0c05002094a0cdbf9cef95a61aeaca921276384"
Nov 02 12:45:17.601430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.601402    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" event={"ID":"18215d47-193b-42f6-9571-957b9b72e581","Type":"ContainerStarted","Data":"c9485fcd8f93bcf4fb9a6c23e9384cbf56bd4f623bd3e95df4330866e5520a88"}
Nov 02 12:45:17.601578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.601453    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" event={"ID":"18215d47-193b-42f6-9571-957b9b72e581","Type":"ContainerStarted","Data":"da7cd510dfd02e44c8960238064fb5d32afe59638bccf4600f7f2260c96e326e"}
Nov 02 12:45:17.665107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.665071    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:17.673024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.672995    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893725-mj5hm"]
Nov 02 12:45:17.709218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:17.709158    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podStartSLOduration=1.709145063 podStartE2EDuration="1.709145063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:17.704617617 +0000 UTC m=+1115.353671576" watchObservedRunningTime="2026-11-02 12:45:17.709145063 +0000 UTC m=+1115.358199020"
Nov 02 12:45:18.369480 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.369447    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:18.371019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.370994    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:45:18.371079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.371051    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 12:45:18.464260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.464221    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:45:18.464985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.464962    4928 cpu_manager.go:414] "containerMap: removing container" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.464985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.464985    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.464999    4928 cpu_manager.go:414] "containerMap: removing container" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465007    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465022    4928 cpu_manager.go:414] "containerMap: removing container" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465029    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e58130-05cf-4096-8f92-e6248eb87722" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465048    4928 cpu_manager.go:414] "containerMap: removing container" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465055    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465103    4928 cpu_manager.go:414] "containerMap: removing container" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465110    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465267    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.465543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465288    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" containerName="cleanup"
Nov 02 12:45:18.465543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.465301    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:18.471134 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.471108    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.490455 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.490426    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:45:18.503149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.503122    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77cf70-dockercfg-x4g8h\""
Nov 02 12:45:18.503317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.503201    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77cf70\""
Nov 02 12:45:18.506172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.506142    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e58130-05cf-4096-8f92-e6248eb87722" path="/var/lib/kubelet/pods/23e58130-05cf-4096-8f92-e6248eb87722/volumes"
Nov 02 12:45:18.506895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.506873    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701ea9e0-a746-4cbf-878d-ce964d40adbe" path="/var/lib/kubelet/pods/701ea9e0-a746-4cbf-878d-ce964d40adbe/volumes"
Nov 02 12:45:18.507488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.507468    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" path="/var/lib/kubelet/pods/798df37d-dd79-4c18-8b33-f3479a609f9b/volumes"
Nov 02 12:45:18.583852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.579645    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pscnh\" (UniqueName: \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.583852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.580571    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.715899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.715185    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pscnh\" (UniqueName: \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.715899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.715395    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.797847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.796785    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pscnh\" (UniqueName: \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:18.858866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:18.856071    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:19.087853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:19.087106    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:19.374906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:19.373776    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:45:19.374906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:19.373861    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 12:45:19.636993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:19.632523    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:45:19.807370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:19.807339    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" event={"ID":"d7bd3535-d8a4-404b-963d-1adeed4b08cd","Type":"ContainerStarted","Data":"4738c2b6cb051ad9ce596512d2714c493322b2d0afb9afaf02635d2851f4e63a"}
Nov 02 12:45:20.373868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.370155    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:45:20.373868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.370220    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 12:45:20.821125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.821088    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" event={"ID":"d7bd3535-d8a4-404b-963d-1adeed4b08cd","Type":"ContainerStarted","Data":"f3102325b67a98ac72ca05eeeb21e71060a9621e2d29667856debfababb3cb30"}
Nov 02 12:45:20.821639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.821320    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:20.882543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.882475    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" podStartSLOduration=2.882455998 podStartE2EDuration="2.882455998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:20.877407058 +0000 UTC m=+1118.526461016" watchObservedRunningTime="2026-11-02 12:45:20.882455998 +0000 UTC m=+1118.531509956"
Nov 02 12:45:20.983147 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.982594    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:45:20.983954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.983924    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="798df37d-dd79-4c18-8b33-f3479a609f9b" containerName="cleanup"
Nov 02 12:45:20.988792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.988763    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:20.989799 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:20.989772    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:45:21.020505 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.020473    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e6ceda-serving-cert\""
Nov 02 12:45:21.102068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.101977    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.102252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.102104    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.102252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.102192    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ck8q\" (UniqueName: \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.102360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.102258    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.204041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.203979    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ck8q\" (UniqueName: \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.204041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.204092    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.204410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.204388    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.204500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.204484    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.205934 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.205903    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.209256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.209221    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.213460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.211901    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.243290 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.238965    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ck8q\" (UniqueName: \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\") pod \"maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.309675 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.309629    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:21.371057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.369164    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:45:21.371057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.369227    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 12:45:21.665707 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.665673    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:45:21.668417 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:45:21.668389    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9881cd_5dac_4cbf_9e84_11133c0c3667.slice/crio-943fe3e239dc47a311c457bfb4ae1ad5f303cfbc7915d83cf8261c16c9cd803e WatchSource:0}: Error finding container 943fe3e239dc47a311c457bfb4ae1ad5f303cfbc7915d83cf8261c16c9cd803e: Status 404 returned error can't find the container with id 943fe3e239dc47a311c457bfb4ae1ad5f303cfbc7915d83cf8261c16c9cd803e
Nov 02 12:45:21.855238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:21.855201    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" event={"ID":"9c9881cd-5dac-4cbf-9e84-11133c0c3667","Type":"ContainerStarted","Data":"943fe3e239dc47a311c457bfb4ae1ad5f303cfbc7915d83cf8261c16c9cd803e"}
Nov 02 12:45:22.371232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:22.371188    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:45:22.371372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:22.371245    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 12:45:22.865555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:22.865519    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" event={"ID":"9c9881cd-5dac-4cbf-9e84-11133c0c3667","Type":"ContainerStarted","Data":"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"}
Nov 02 12:45:22.866002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:22.865621    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:22.946863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:22.944434    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" podStartSLOduration=2.429000095 podStartE2EDuration="2.944409691s" totalImagesPullingTime="515.409596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:22.944073209 +0000 UTC m=+1120.593127169" watchObservedRunningTime="2026-11-02 12:45:22.944409691 +0000 UTC m=+1120.593463650"
Nov 02 12:45:23.374250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:23.374216    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:23.375162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:23.374743    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:23.375714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:23.375687    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:45:28.881327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:28.881288    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:45:31.858580 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:31.858550    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:45:39.739988 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:45:39.739943    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:45:44.125142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.125109    4928 generic.go:356] "Generic (PLEG): container finished" podID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerID="aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585" exitCode=0
Nov 02 12:45:44.125583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.125148    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerDied","Data":"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"}
Nov 02 12:45:44.125583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.125559    4928 scope.go:122] "RemoveContainer" containerID="aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"
Nov 02 12:45:44.127183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.127148    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:44.127275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.127219    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:44.127275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:44.127250    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:45:45.158610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:45.157522    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerStarted","Data":"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"}
Nov 02 12:45:45.248613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:45.248204    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:45.248613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:45.248320    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:45.248613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:45.248360    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:46.249851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:46.248701    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:46.249851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:46.248761    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:47.252468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:47.252415    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:47.253045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:47.252492    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:48.249876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:48.249603    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:48.249876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:48.249672    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:49.250150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:49.248208    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:49.250150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:49.248271    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:50.243236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:50.243192    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:50.246126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:50.246095    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 12:45:50.246260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:50.246158    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 12:45:51.251080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:51.251044    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:51.252286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:51.252260    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:45:57.109715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:45:57.109677    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:46:01.183137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.183094    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:46:01.188019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.187991    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.212026 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.211997    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:46:01.260207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.260169    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.260503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.260476    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.260683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.260663    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.260821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.260803    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.261008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.260990    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.261133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.261117    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.261339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.261322    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.261479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.261463    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79zt\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.261734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.261713    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.363942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.363896    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364131 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.363990    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364131 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364058    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364131 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364111    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364137    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364200    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364225    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364348    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364382    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79zt\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.364605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.364578    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.365058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.365029    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.365416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.365391    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.365506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.365414    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.365617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.365596    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.366606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.366579    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.366963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.366942    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.392705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.392678    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.392871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.392812    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79zt\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.506444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.506406    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:01.843408 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.843372    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:46:01.851259 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.851216    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:01.852216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.851316    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:01.852216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:01.851363    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:02.110765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.110667    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" podUID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" containerName="payload-pre-processing" containerID="cri-o://f3102325b67a98ac72ca05eeeb21e71060a9621e2d29667856debfababb3cb30" gracePeriod=25
Nov 02 12:46:02.418053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.417994    4928 generic.go:356] "Generic (PLEG): container finished" podID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" containerID="f3102325b67a98ac72ca05eeeb21e71060a9621e2d29667856debfababb3cb30" exitCode=0
Nov 02 12:46:02.418495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.418336    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" event={"ID":"d7bd3535-d8a4-404b-963d-1adeed4b08cd","Type":"ContainerDied","Data":"f3102325b67a98ac72ca05eeeb21e71060a9621e2d29667856debfababb3cb30"}
Nov 02 12:46:02.418495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.418396    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4" event={"ID":"d7bd3535-d8a4-404b-963d-1adeed4b08cd","Type":"ContainerDied","Data":"4738c2b6cb051ad9ce596512d2714c493322b2d0afb9afaf02635d2851f4e63a"}
Nov 02 12:46:02.418495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.418421    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4738c2b6cb051ad9ce596512d2714c493322b2d0afb9afaf02635d2851f4e63a"
Nov 02 12:46:02.420708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.420679    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" event={"ID":"cd446b0c-2951-4da4-9d28-6cdfec83dbaa","Type":"ContainerStarted","Data":"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"}
Nov 02 12:46:02.420860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.420734    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" event={"ID":"cd446b0c-2951-4da4-9d28-6cdfec83dbaa","Type":"ContainerStarted","Data":"d3b52089722f13efd75721cc7619e362c4e10a6eaec1bdb237f283e7b091a742"}
Nov 02 12:46:02.438195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.438166    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:46:02.468474 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.468411    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podStartSLOduration=1.468393395 podStartE2EDuration="1.468393395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:02.467596817 +0000 UTC m=+1160.116650788" watchObservedRunningTime="2026-11-02 12:46:02.468393395 +0000 UTC m=+1160.117447354"
Nov 02 12:46:02.507030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.507003    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:02.509020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.508994    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:02.509216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.509045    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:02.595912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.595878    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\" (UniqueName: \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\") pod \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") "
Nov 02 12:46:02.596083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.595952    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\") pod \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\" (UID: \"d7bd3535-d8a4-404b-963d-1adeed4b08cd\") "
Nov 02 12:46:02.596334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.596300    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume" pod "d7bd3535-d8a4-404b-963d-1adeed4b08cd" (UID: "d7bd3535-d8a4-404b-963d-1adeed4b08cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:02.597021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.596999    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7bd3535-d8a4-404b-963d-1adeed4b08cd-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:02.598296 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.598262    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh" pod "d7bd3535-d8a4-404b-963d-1adeed4b08cd" (UID: "d7bd3535-d8a4-404b-963d-1adeed4b08cd"). InnerVolumeSpecName "kube-api-access-pscnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:02.699495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:02.699386    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pscnh\" (UniqueName: \"kubernetes.io/projected/d7bd3535-d8a4-404b-963d-1adeed4b08cd-kube-api-access-pscnh\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:03.449567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:03.449526    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"
Nov 02 12:46:03.512277 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:03.512239    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:03.512566 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:03.512537    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:03.596959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:03.596914    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:46:03.615771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:03.614638    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-7b6c5d49dc-8rhg4"]
Nov 02 12:46:04.499095 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:04.499057    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" path="/var/lib/kubelet/pods/d7bd3535-d8a4-404b-963d-1adeed4b08cd/volumes"
Nov 02 12:46:04.507464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:04.507432    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:04.507611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:04.507489    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:05.506963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:05.506914    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:05.507511 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:05.506982    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:06.509867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:06.506945    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:06.509867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:06.507006    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:07.054936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:07.054893    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"]
Nov 02 12:46:07.065401 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:07.065365    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6efeb9-5xp28"]
Nov 02 12:46:07.507032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:07.506999    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 12:46:07.507205 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:07.507049    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 12:46:08.502103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:08.502045    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0377b40-6d16-4b21-a005-914f934788cb" path="/var/lib/kubelet/pods/c0377b40-6d16-4b21-a005-914f934788cb/volumes"
Nov 02 12:46:08.515414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:08.515371    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:08.516350 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:08.516314    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:08.517218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:08.517194    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:46:09.732159 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:09.732111    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:46:18.401105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:18.401060    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:46:18.401603 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:18.401359    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy" containerID="cri-o://faff8c715e8e66acfe4a9b6ea73a4a01ddd0a28b297e90514f2664c7cefef83f" gracePeriod=30
Nov 02 12:46:21.267818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.267775    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:46:21.268875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.268849    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" containerName="payload-pre-processing"
Nov 02 12:46:21.268875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.268875    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" containerName="payload-pre-processing"
Nov 02 12:46:21.269054 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.269037    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bd3535-d8a4-404b-963d-1adeed4b08cd" containerName="payload-pre-processing"
Nov 02 12:46:21.273461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.273438    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.292077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.292054    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:46:21.299783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.299752    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-97de36\""
Nov 02 12:46:21.315854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.315807    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thsv\" (UniqueName: \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.316174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.316149    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.421621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.421562    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.422047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.422019    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thsv\" (UniqueName: \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.423076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.423049    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.507442 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.507032    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thsv\" (UniqueName: \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\") pod \"payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.593186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.593083    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:21.987631 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:21.987599    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542363cc_8598_4b64_950d_1b0fff4db6bb.slice/crio-a171131270e64732ebb2dcb3adef04a5e8813ede31da0ee19639c488ce0169b1 WatchSource:0}: Error finding container a171131270e64732ebb2dcb3adef04a5e8813ede31da0ee19639c488ce0169b1: Status 404 returned error can't find the container with id a171131270e64732ebb2dcb3adef04a5e8813ede31da0ee19639c488ce0169b1
Nov 02 12:46:21.987794 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:21.987737    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:46:22.690937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:22.690899    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" event={"ID":"542363cc-8598-4b64-950d-1b0fff4db6bb","Type":"ContainerStarted","Data":"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"}
Nov 02 12:46:22.691340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:22.690957    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" event={"ID":"542363cc-8598-4b64-950d-1b0fff4db6bb","Type":"ContainerStarted","Data":"a171131270e64732ebb2dcb3adef04a5e8813ede31da0ee19639c488ce0169b1"}
Nov 02 12:46:22.691340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:22.691097    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:22.738531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:22.738462    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" podStartSLOduration=1.7384450089999999 podStartE2EDuration="1.738445009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:22.734054071 +0000 UTC m=+1180.383108029" watchObservedRunningTime="2026-11-02 12:46:22.738445009 +0000 UTC m=+1180.387498968"
Nov 02 12:46:23.710990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.710952    4928 generic.go:356] "Generic (PLEG): container finished" podID="82e748a4-4a4f-4087-a511-d6aa84662287" containerID="faff8c715e8e66acfe4a9b6ea73a4a01ddd0a28b297e90514f2664c7cefef83f" exitCode=0
Nov 02 12:46:23.711381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.711026    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerDied","Data":"faff8c715e8e66acfe4a9b6ea73a4a01ddd0a28b297e90514f2664c7cefef83f"}
Nov 02 12:46:23.711381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.711099    4928 scope.go:122] "RemoveContainer" containerID="7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c"
Nov 02 12:46:23.738544 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:23.738502    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c\": container with ID starting with 7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c not found: ID does not exist" containerID="7811eaa90e8d84bb620b8ac8b4dc69451b3c5114c9ec2d415e8ad2d90c0d380c"
Nov 02 12:46:23.742539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.742495    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:46:23.810346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.810305    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.810625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.810607    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.815447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.815416    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.822730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.822700    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.822993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.822963    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.823278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.811296    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:23.823357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823237    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:23.823357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823264    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.823544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823510    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.823727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823697    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:23.823727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823718    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.823897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.823804    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\") pod \"82e748a4-4a4f-4087-a511-d6aa84662287\" (UID: \"82e748a4-4a4f-4087-a511-d6aa84662287\") "
Nov 02 12:46:23.824104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.824058    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:23.824245 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.824223    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:23.825794 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.825770    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.825986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.825969    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.826087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.826074    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.826178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.826167    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.826265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.826254    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e748a4-4a4f-4087-a511-d6aa84662287-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.829337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.829313    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:23.838033 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.834997    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:23.841057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.841011    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:23.884078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.874464    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8" pod "82e748a4-4a4f-4087-a511-d6aa84662287" (UID: "82e748a4-4a4f-4087-a511-d6aa84662287"). InnerVolumeSpecName "kube-api-access-wvhx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:23.932740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.931263    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e748a4-4a4f-4087-a511-d6aa84662287-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.932740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.931298    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.932740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.931312    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e748a4-4a4f-4087-a511-d6aa84662287-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:23.932740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:23.931325    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhx8\" (UniqueName: \"kubernetes.io/projected/82e748a4-4a4f-4087-a511-d6aa84662287-kube-api-access-wvhx8\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.720537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:24.720446    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"
Nov 02 12:46:24.720537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:24.720475    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv" event={"ID":"82e748a4-4a4f-4087-a511-d6aa84662287","Type":"ContainerDied","Data":"e4e51a9ce62859f456c3ab8ba4188389b58d450981ba4cf4231050e7c6c8f1f8"}
Nov 02 12:46:24.720537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:24.720532    4928 scope.go:122] "RemoveContainer" containerID="faff8c715e8e66acfe4a9b6ea73a4a01ddd0a28b297e90514f2664c7cefef83f"
Nov 02 12:46:24.793985 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:24.793157    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:46:24.805393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:24.805358    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-cvshv"]
Nov 02 12:46:25.547044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.547002    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:46:25.547769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.547747    4928 cpu_manager.go:414] "containerMap: removing container" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:25.547769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.547770    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:25.547980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.547804    4928 cpu_manager.go:414] "containerMap: removing container" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:25.547980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.547812    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:25.548083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.548066    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:25.553446 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.553424    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.570091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.570063    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:46:25.576883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.576857    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b2ec05-dockercfg-wcx4n\""
Nov 02 12:46:25.616729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.616695    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b2ec05\""
Nov 02 12:46:25.675202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.675161    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.675717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.675541    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglzr\" (UniqueName: \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.777586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.777528    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglzr\" (UniqueName: \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.778262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.778239    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.780613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.779310    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.814900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.804164    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglzr\" (UniqueName: \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\") pod \"payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:25.872849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:25.872137    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:26.259504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.259469    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:46:26.260739 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:26.260709    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bfdb1a_a95a_460c_a669_91ad43ad96d3.slice/crio-74bba68d9dce4745e8f693af9694f97776ea0100bd60ffe28f451085eae2f77f WatchSource:0}: Error finding container 74bba68d9dce4745e8f693af9694f97776ea0100bd60ffe28f451085eae2f77f: Status 404 returned error can't find the container with id 74bba68d9dce4745e8f693af9694f97776ea0100bd60ffe28f451085eae2f77f
Nov 02 12:46:26.499478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.499444    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" path="/var/lib/kubelet/pods/82e748a4-4a4f-4087-a511-d6aa84662287/volumes"
Nov 02 12:46:26.746653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.746614    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" event={"ID":"34bfdb1a-a95a-460c-a669-91ad43ad96d3","Type":"ContainerStarted","Data":"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"}
Nov 02 12:46:26.746918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.746675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" event={"ID":"34bfdb1a-a95a-460c-a669-91ad43ad96d3","Type":"ContainerStarted","Data":"74bba68d9dce4745e8f693af9694f97776ea0100bd60ffe28f451085eae2f77f"}
Nov 02 12:46:26.746918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.746727    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:26.792511 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:26.792456    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" podStartSLOduration=1.792440196 podStartE2EDuration="1.792440196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:26.787955357 +0000 UTC m=+1184.437009318" watchObservedRunningTime="2026-11-02 12:46:26.792440196 +0000 UTC m=+1184.441494153"
Nov 02 12:46:29.682888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.682853    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:46:29.683765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.683741    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e748a4-4a4f-4087-a511-d6aa84662287" containerName="istio-proxy"
Nov 02 12:46:29.687802 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.687782    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.714243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.714213    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-729108-openshift-default-dockercfg-zxwhv\""
Nov 02 12:46:29.734542 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.734512    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:46:29.744960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.744911    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745038    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745083    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745215    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745278    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745335    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhdf\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745369    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745396    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.745519 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.745444    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.867850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.846911    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.868248 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.868217    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.868409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.868394    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.868571 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.868557    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.868738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.868697    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.868910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.868890    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhdf\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.869064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.869046    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.869179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.869162    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.869320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.869302    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.869996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.869971    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.872646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.871083    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.879902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.863773    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.880147 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.871563    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.880402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.872583    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.933725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.892015    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.933725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.911108    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhdf\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.941255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.934858    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:29.958854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:29.947744    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\") pod \"e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") " pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:30.017052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.017016    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:30.367417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.367387    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:46:30.369452 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:30.369422    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c57e715_0797_4a0d_ab7f_6e64b0e4d71c.slice/crio-e2fd63ac5b04e8d81e51467f379bc051e88fedbfa27e012a447badad4750b868 WatchSource:0}: Error finding container e2fd63ac5b04e8d81e51467f379bc051e88fedbfa27e012a447badad4750b868: Status 404 returned error can't find the container with id e2fd63ac5b04e8d81e51467f379bc051e88fedbfa27e012a447badad4750b868
Nov 02 12:46:30.372291 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.372255    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:30.372405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.372321    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:30.372405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.372350    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:30.837908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.837859    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerStarted","Data":"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"}
Nov 02 12:46:30.838386 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.837961    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerStarted","Data":"e2fd63ac5b04e8d81e51467f379bc051e88fedbfa27e012a447badad4750b868"}
Nov 02 12:46:30.888580 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:30.888515    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podStartSLOduration=1.8884961489999998 podStartE2EDuration="1.888496149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:30.886303635 +0000 UTC m=+1188.535357595" watchObservedRunningTime="2026-11-02 12:46:30.888496149 +0000 UTC m=+1188.537550123"
Nov 02 12:46:31.026852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:31.025919    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:31.046900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:31.046281    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:31.845482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:31.845450    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:31.846596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:31.846571    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:46:33.714472 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:33.714441    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:46:37.758993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:37.758953    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:46:39.056718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.056679    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:46:39.063468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.063438    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.085182 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.082390    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:46:39.089039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.089012    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-078fd3-openshift-default-dockercfg-z2lq7\""
Nov 02 12:46:39.263323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.263281    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.263656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.263634    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.263942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.263924    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.264178 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.264161    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.264351 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.264335    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.264524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.264508    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.264713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.264695    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czq79\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.269848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.264885    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.270069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.270045    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.375159 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.375051    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.376414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.376388    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.376954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.376935    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.377445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.377426    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.377604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.377590    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czq79\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.377746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.377733    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.388855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.377838    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.389170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.389143    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.389304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.389290    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.389528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.376864    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.389660 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.377334    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.395848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.390048    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.396010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.376225    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.396106 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.394437    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.396207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.394994    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.420612 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.420580    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czq79\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.420902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.420879    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.427133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.427096    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\") pod \"e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") " pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.696142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:39.696040    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:39.755520 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:39.755419    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:46:40.061915 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.061809    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:46:40.062423 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:40.062140    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95641b25_f013_40d4_bfae_35c996deea42.slice/crio-be58e8e34003fdaaadc8016cec53e06fe24a23087ace00a520df2a3571d2bbdf WatchSource:0}: Error finding container be58e8e34003fdaaadc8016cec53e06fe24a23087ace00a520df2a3571d2bbdf: Status 404 returned error can't find the container with id be58e8e34003fdaaadc8016cec53e06fe24a23087ace00a520df2a3571d2bbdf
Nov 02 12:46:40.065013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.064975    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:40.065105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.065046    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:40.065105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.065076    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:46:40.939984 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.939932    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerStarted","Data":"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"}
Nov 02 12:46:40.940201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:40.940042    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerStarted","Data":"be58e8e34003fdaaadc8016cec53e06fe24a23087ace00a520df2a3571d2bbdf"}
Nov 02 12:46:41.017758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:41.017691    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podStartSLOduration=2.017668743 podStartE2EDuration="2.017668743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:41.00194639 +0000 UTC m=+1198.651000351" watchObservedRunningTime="2026-11-02 12:46:41.017668743 +0000 UTC m=+1198.666722702"
Nov 02 12:46:41.697331 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:41.697296    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:41.703340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:41.703312    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:41.949177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:41.949082    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:41.950097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:41.950073    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:46:42.747392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:42.747356    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:46:42.750726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:42.750697    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:46:43.146874 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:43.146772    4928 scope.go:122] "RemoveContainer" containerID="dfd8ae6b607bf2320f345a29151fcde4f00e9b9220b8156971fd04af4bc75169"
Nov 02 12:46:43.162602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:43.162577    4928 scope.go:122] "RemoveContainer" containerID="8fd2bea99df201980c32dbf843f35de6183aa498eee900e5a61beba22063e991"
Nov 02 12:46:45.801481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:45.801447    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:46:45.810003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:45.809969    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:45.842626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:45.840314    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:46:45.851611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:45.851378    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-078fd3\""
Nov 02 12:46:45.851611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:45.851557    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-078fd3-dockercfg-6bkf8\""
Nov 02 12:46:46.037873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.037810    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:46:46.050176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.050129    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:46:46.050419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.050392    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.054036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.053938    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ts6l\" (UniqueName: \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.055275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.055230    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.175868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.173304    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ts6l\" (UniqueName: \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.175868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.173577    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.175868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.174422    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.203617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.203588    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ts6l\" (UniqueName: \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.222078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.222032    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:46.277223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.277160    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.277441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.277271    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65j74\" (UniqueName: \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.379233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.379154    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.379233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.379216    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65j74\" (UniqueName: \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.379983 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.379959    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.404295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.404262    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65j74\" (UniqueName: \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\") pod \"payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.482431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.482392    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:46.597344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.596970    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:46:46.610654 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:46.610613    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c194de8_33b5_4698_bc0e_a80db65055db.slice/crio-02144e6875d85a2e38e7260f3f93e560c73aef99be0cd54c720a03bb2d687707 WatchSource:0}: Error finding container 02144e6875d85a2e38e7260f3f93e560c73aef99be0cd54c720a03bb2d687707: Status 404 returned error can't find the container with id 02144e6875d85a2e38e7260f3f93e560c73aef99be0cd54c720a03bb2d687707
Nov 02 12:46:46.800567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:46.800526    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:46:46.800905 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:46:46.800878    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ab9e0c_f754_4784_a1f3_848211e58191.slice/crio-3ea64c511881e7d50c820a93cb9b86ee80bcb853ac57681d86516ab68cb5828e WatchSource:0}: Error finding container 3ea64c511881e7d50c820a93cb9b86ee80bcb853ac57681d86516ab68cb5828e: Status 404 returned error can't find the container with id 3ea64c511881e7d50c820a93cb9b86ee80bcb853ac57681d86516ab68cb5828e
Nov 02 12:46:47.196078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.196036    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" event={"ID":"1c194de8-33b5-4698-bc0e-a80db65055db","Type":"ContainerStarted","Data":"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"}
Nov 02 12:46:47.196731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.196109    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" event={"ID":"1c194de8-33b5-4698-bc0e-a80db65055db","Type":"ContainerStarted","Data":"02144e6875d85a2e38e7260f3f93e560c73aef99be0cd54c720a03bb2d687707"}
Nov 02 12:46:47.196731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.196163    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:46:47.198145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.198108    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" event={"ID":"b0ab9e0c-f754-4784-a1f3-848211e58191","Type":"ContainerStarted","Data":"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"}
Nov 02 12:46:47.198302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.198166    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" event={"ID":"b0ab9e0c-f754-4784-a1f3-848211e58191","Type":"ContainerStarted","Data":"3ea64c511881e7d50c820a93cb9b86ee80bcb853ac57681d86516ab68cb5828e"}
Nov 02 12:46:47.198302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.198210    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:47.253317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.253262    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" podStartSLOduration=2.253248626 podStartE2EDuration="2.253248626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:47.246974537 +0000 UTC m=+1204.896028496" watchObservedRunningTime="2026-11-02 12:46:47.253248626 +0000 UTC m=+1204.902302617"
Nov 02 12:46:47.302332 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:47.302277    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" podStartSLOduration=2.3022621 podStartE2EDuration="2.3022621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:47.299467767 +0000 UTC m=+1204.948521727" watchObservedRunningTime="2026-11-02 12:46:47.3022621 +0000 UTC m=+1204.951316058"
Nov 02 12:46:48.183318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.183284    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:46:48.183510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.183475    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy" containerID="cri-o://6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb" gracePeriod=30
Nov 02 12:46:48.197162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.197128    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:46:48.197631 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.197372    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy" containerID="cri-o://2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d" gracePeriod=30
Nov 02 12:46:48.202478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.202451    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:46:48.202636 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.202616    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy" containerID="cri-o://8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789" gracePeriod=30
Nov 02 12:46:48.212634 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.212605    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:46:48.212849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:48.212804    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy" containerID="cri-o://91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13" gracePeriod=30
Nov 02 12:46:53.613702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.612966    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:46:53.643489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.643399    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:46:53.754049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.753979    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:46:53.759315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.759070    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:46:53.766657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.766634    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.766797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.766683    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.766797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.766729    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.766943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.766813    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767116    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767156    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767147    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767311    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767334    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.767346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767341    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767435    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767474    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767544    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767578    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767632    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767670    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767675    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767749    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767778    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767881    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767898    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.767918    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\") pod \"48e8696c-fe05-45a5-90d3-b4343d48bda2\" (UID: \"48e8696c-fe05-45a5-90d3-b4343d48bda2\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768040    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768093    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\") pod \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\" (UID: \"816eae63-ce87-48e5-84d7-bfeec7c63a6c\") "
Nov 02 12:46:53.768214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768100    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.768890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768868    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768894    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768911    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768925    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768940    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768954    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.768966    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e8696c-fe05-45a5-90d3-b4343d48bda2-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.769363 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.769095    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.769363 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.769130    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.769708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.769683    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:53.771521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.771500    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:53.771620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.771596    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.778556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.778531    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.778662 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.778548    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "kube-api-access-h4std". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.778662 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.778535    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.778922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.778889    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.780003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.779967    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo" pod "48e8696c-fe05-45a5-90d3-b4343d48bda2" (UID: "48e8696c-fe05-45a5-90d3-b4343d48bda2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:53.780409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.780386    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq" pod "816eae63-ce87-48e5-84d7-bfeec7c63a6c" (UID: "816eae63-ce87-48e5-84d7-bfeec7c63a6c"). InnerVolumeSpecName "kube-api-access-prjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.869589 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869530    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.869778 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869624    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.869778 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869738    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.869933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869785    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.869933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869859    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.869933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869905    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.870101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869937    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.869990    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.870101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870021    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870051    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870092    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870153    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870212    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870244    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.870417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870311    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.870417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870343    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.870681 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870645    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.870881 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.870859    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.872547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.872056    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:53.872547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.872328    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.873116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.873010    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.873975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.873201    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.873975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.873396    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.873975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.873688    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:53.875020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.874982    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\") pod \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\" (UID: \"c5b40528-7325-49d7-9b2d-edf9c0997dd6\") "
Nov 02 12:46:53.875277 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.875255    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\") pod \"d58dba09-86a0-4bdd-b0a1-ef1431850452\" (UID: \"d58dba09-86a0-4bdd-b0a1-ef1431850452\") "
Nov 02 12:46:53.876261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876238    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876386 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876373    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876462    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prjcq\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-kube-api-access-prjcq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876544    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876626    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876702    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876781    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.876922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876911    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58dba09-86a0-4bdd-b0a1-ef1431850452-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.876993    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877074    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e8696c-fe05-45a5-90d3-b4343d48bda2-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877150    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877225    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4std\" (UniqueName: \"kubernetes.io/projected/48e8696c-fe05-45a5-90d3-b4343d48bda2-kube-api-access-h4std\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877308    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877391 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877381    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877474 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877465    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877551 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877541    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877628    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877778 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877767    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.877883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.877873    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816eae63-ce87-48e5-84d7-bfeec7c63a6c-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.882617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.882371    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.882617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.882449    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.882617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.882493    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "kube-api-access-hg8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.883345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.882926    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.883345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.882994    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:53.883345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.883227    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.883652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.883619    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:53.884145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.884112    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token" pod "c5b40528-7325-49d7-9b2d-edf9c0997dd6" (UID: "c5b40528-7325-49d7-9b2d-edf9c0997dd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.884301 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.884287    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:53.887886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.886840    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd" pod "d58dba09-86a0-4bdd-b0a1-ef1431850452" (UID: "d58dba09-86a0-4bdd-b0a1-ef1431850452"). InnerVolumeSpecName "kube-api-access-gtdqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:53.979539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979504    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979537    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979546    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979557    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979566    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8bb\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-kube-api-access-hg8bb\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979578    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979587    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b40528-7325-49d7-9b2d-edf9c0997dd6-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979596    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b40528-7325-49d7-9b2d-edf9c0997dd6-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979604    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:53.979779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:53.979624    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtdqd\" (UniqueName: \"kubernetes.io/projected/d58dba09-86a0-4bdd-b0a1-ef1431850452-kube-api-access-gtdqd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:46:54.284738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.284701    4928 generic.go:356] "Generic (PLEG): container finished" podID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerID="8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789" exitCode=0
Nov 02 12:46:54.284902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.284743    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"
Nov 02 12:46:54.284902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.284783    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" event={"ID":"48e8696c-fe05-45a5-90d3-b4343d48bda2","Type":"ContainerDied","Data":"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"}
Nov 02 12:46:54.284902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.284857    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws" event={"ID":"48e8696c-fe05-45a5-90d3-b4343d48bda2","Type":"ContainerDied","Data":"489dc32cf367978e8ba8974ec17798e8ff848a2a2a0d5c986a03a28023c199c6"}
Nov 02 12:46:54.284902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.284904    4928 scope.go:122] "RemoveContainer" containerID="8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"
Nov 02 12:46:54.287545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.287518    4928 generic.go:356] "Generic (PLEG): container finished" podID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerID="2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d" exitCode=0
Nov 02 12:46:54.287683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.287557    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"
Nov 02 12:46:54.287683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.287595    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerDied","Data":"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"}
Nov 02 12:46:54.287683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.287658    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9" event={"ID":"d58dba09-86a0-4bdd-b0a1-ef1431850452","Type":"ContainerDied","Data":"1dbd153c3be291b195073dc903e1a6a42f0ae10526f8785f65a61029d7348692"}
Nov 02 12:46:54.292249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.292225    4928 generic.go:356] "Generic (PLEG): container finished" podID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerID="91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13" exitCode=0
Nov 02 12:46:54.292370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.292286    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"
Nov 02 12:46:54.292370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.292308    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" event={"ID":"c5b40528-7325-49d7-9b2d-edf9c0997dd6","Type":"ContainerDied","Data":"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"}
Nov 02 12:46:54.292370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.292350    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r" event={"ID":"c5b40528-7325-49d7-9b2d-edf9c0997dd6","Type":"ContainerDied","Data":"ecd85c2aeee344cfa7d79c2286bdfc981ec6e3501c1fe66265c3db21edee69ee"}
Nov 02 12:46:54.298126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.295888    4928 generic.go:356] "Generic (PLEG): container finished" podID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerID="6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb" exitCode=0
Nov 02 12:46:54.298126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.296041    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" event={"ID":"816eae63-ce87-48e5-84d7-bfeec7c63a6c","Type":"ContainerDied","Data":"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"}
Nov 02 12:46:54.298126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.296077    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf" event={"ID":"816eae63-ce87-48e5-84d7-bfeec7c63a6c","Type":"ContainerDied","Data":"008ac08befeedf4aa7586dd03291bce64e7943c3a9b82d1a1aa6d197ad5e43d4"}
Nov 02 12:46:54.298126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.296158    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"
Nov 02 12:46:54.302695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.302675    4928 scope.go:122] "RemoveContainer" containerID="8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"
Nov 02 12:46:54.303015 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:54.302986    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789\": container with ID starting with 8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789 not found: ID does not exist" containerID="8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"
Nov 02 12:46:54.303097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.303026    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789"} err="failed to get container status \"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789\": rpc error: code = NotFound desc = could not find container \"8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789\": container with ID starting with 8559ca38e60374ae3001dd2e41d83ba146e6fa14a43da2788afa885666472789 not found: ID does not exist"
Nov 02 12:46:54.303097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.303048    4928 scope.go:122] "RemoveContainer" containerID="2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"
Nov 02 12:46:54.315525 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.315507    4928 scope.go:122] "RemoveContainer" containerID="6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"
Nov 02 12:46:54.326135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.325909    4928 scope.go:122] "RemoveContainer" containerID="2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"
Nov 02 12:46:54.326247 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:54.326215    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d\": container with ID starting with 2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d not found: ID does not exist" containerID="2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"
Nov 02 12:46:54.326308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.326254    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d"} err="failed to get container status \"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d\": rpc error: code = NotFound desc = could not find container \"2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d\": container with ID starting with 2568529fa2812c43b59c991560120622c84265283234e2de1ddb6b429ea7440d not found: ID does not exist"
Nov 02 12:46:54.326308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.326272    4928 scope.go:122] "RemoveContainer" containerID="6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"
Nov 02 12:46:54.326579 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:54.326552    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4\": container with ID starting with 6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4 not found: ID does not exist" containerID="6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"
Nov 02 12:46:54.326651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.326596    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4"} err="failed to get container status \"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4\": rpc error: code = NotFound desc = could not find container \"6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4\": container with ID starting with 6431e63b0cf5e9f426ff698bba4b2e125d478dcbf98db437a8a3eead2f0f8ac4 not found: ID does not exist"
Nov 02 12:46:54.326651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.326620    4928 scope.go:122] "RemoveContainer" containerID="91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"
Nov 02 12:46:54.339562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.339538    4928 scope.go:122] "RemoveContainer" containerID="91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"
Nov 02 12:46:54.339921 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:54.339898    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13\": container with ID starting with 91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13 not found: ID does not exist" containerID="91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"
Nov 02 12:46:54.340015 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.339930    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13"} err="failed to get container status \"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13\": rpc error: code = NotFound desc = could not find container \"91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13\": container with ID starting with 91ccbe89366ab90414265e9188459006ee05c1a741a2c46bf157c35120bc4c13 not found: ID does not exist"
Nov 02 12:46:54.340015 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.339952    4928 scope.go:122] "RemoveContainer" containerID="6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"
Nov 02 12:46:54.351574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.351557    4928 scope.go:122] "RemoveContainer" containerID="6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"
Nov 02 12:46:54.351867 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:46:54.351819    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb\": container with ID starting with 6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb not found: ID does not exist" containerID="6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"
Nov 02 12:46:54.351931 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.351871    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb"} err="failed to get container status \"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb\": rpc error: code = NotFound desc = could not find container \"6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb\": container with ID starting with 6577218f1c9219e3951f2d2ab93dde0deae13207491428f9aa7a8c2a3a5b53eb not found: ID does not exist"
Nov 02 12:46:54.362218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.362190    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:46:54.376607 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.376582    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-4qdk9"]
Nov 02 12:46:54.401562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.401532    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:46:54.408752 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.408725    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-v2xjf"]
Nov 02 12:46:54.445866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.445793    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:46:54.449730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.449697    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qf7ws"]
Nov 02 12:46:54.486002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.485950    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:46:54.499369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.499340    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" path="/var/lib/kubelet/pods/48e8696c-fe05-45a5-90d3-b4343d48bda2/volumes"
Nov 02 12:46:54.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.499884    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" path="/var/lib/kubelet/pods/816eae63-ce87-48e5-84d7-bfeec7c63a6c/volumes"
Nov 02 12:46:54.500367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.500353    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" path="/var/lib/kubelet/pods/d58dba09-86a0-4bdd-b0a1-ef1431850452/volumes"
Nov 02 12:46:54.501042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:54.501026    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-vg72r"]
Nov 02 12:46:55.502296 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.502259    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:46:55.503094 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503075    4928 cpu_manager.go:414] "containerMap: removing container" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503095    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503108    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503113    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503128    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503135    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503151    4928 cpu_manager.go:414] "containerMap: removing container" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy"
Nov 02 12:46:55.503163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503156    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy"
Nov 02 12:46:55.503367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503252    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:46:55.503367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503262    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="816eae63-ce87-48e5-84d7-bfeec7c63a6c" containerName="istio-proxy"
Nov 02 12:46:55.503367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503275    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e8696c-fe05-45a5-90d3-b4343d48bda2" containerName="istio-proxy"
Nov 02 12:46:55.503367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.503282    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" containerName="istio-proxy"
Nov 02 12:46:55.508624 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.508605    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.548844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.541901    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:46:55.589573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.589318    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-729108-serving-cert\""
Nov 02 12:46:55.600463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.600431    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.600627 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.600607    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.600770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.600752    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.600807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.600788    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhk86\" (UniqueName: \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.703372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.703313    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.703687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.703664    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhk86\" (UniqueName: \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.704015 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.703871    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.704370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.704347    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.715851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.709073    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.718945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.718247    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.723237 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.722715    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.741850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.735991    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhk86\" (UniqueName: \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\") pod \"maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:55.827557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:55.827464    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:56.167387 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:56.167358    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:46:56.174427 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:56.174397    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:46:56.323805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:56.323769    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" event={"ID":"f70d5748-9c03-4014-994c-e8c3bc97ecd9","Type":"ContainerStarted","Data":"e18bfd06f51d7d30fa019e253368c1683b3120547c78a29d75465d56b2f93531"}
Nov 02 12:46:56.502024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:56.501989    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b40528-7325-49d7-9b2d-edf9c0997dd6" path="/var/lib/kubelet/pods/c5b40528-7325-49d7-9b2d-edf9c0997dd6/volumes"
Nov 02 12:46:57.334704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:57.334666    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" event={"ID":"f70d5748-9c03-4014-994c-e8c3bc97ecd9","Type":"ContainerStarted","Data":"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"}
Nov 02 12:46:57.335249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:57.334803    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:46:57.390185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:57.386897    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" podStartSLOduration=2.042267206 podStartE2EDuration="2.386814168s" totalImagesPullingTime="344.546962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:57.379294695 +0000 UTC m=+1215.028348654" watchObservedRunningTime="2026-11-02 12:46:57.386814168 +0000 UTC m=+1215.035868145"
Nov 02 12:46:58.214416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:58.214381    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:46:58.214606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:46:58.214459    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:47:03.359138 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:03.359084    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:47:04.434577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.434537    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:47:04.435146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.434799    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" podUID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" containerName="maas-api" containerID="cri-o://776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c" gracePeriod=30
Nov 02 12:47:04.798318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.798284    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:47:04.904749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.904702    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\") pod \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") "
Nov 02 12:47:04.904959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.904923    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\") pod \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") "
Nov 02 12:47:04.905090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.905059    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\") pod \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") "
Nov 02 12:47:04.905166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.905098    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\" (UniqueName: \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\") pod \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\" (UID: \"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e\") "
Nov 02 12:47:04.905975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.905882    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca" pod "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" (UID: "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:04.907953 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.907927    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs" pod "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" (UID: "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:47:04.908176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.908144    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls" pod "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" (UID: "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:47:04.908409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:04.908390    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5" pod "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" (UID: "7983f6ed-cfd2-43e1-85ec-284d6c6cd37e"). InnerVolumeSpecName "kube-api-access-mzgl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:05.011852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.009096    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.011852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.009140    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.011852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.009156    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzgl5\" (UniqueName: \"kubernetes.io/projected/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-kube-api-access-mzgl5\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.011852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.009171    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.424798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.424767    4928 generic.go:356] "Generic (PLEG): container finished" podID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" containerID="776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c" exitCode=0
Nov 02 12:47:05.425002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.424866    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"
Nov 02 12:47:05.425002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.424864    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" event={"ID":"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e","Type":"ContainerDied","Data":"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"}
Nov 02 12:47:05.425081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.425036    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5" event={"ID":"7983f6ed-cfd2-43e1-85ec-284d6c6cd37e","Type":"ContainerDied","Data":"fe323a64eee073dc56844a7c023d34d9ad94453926ead96f46d3b8c3749c20dc"}
Nov 02 12:47:05.425081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.425067    4928 scope.go:122] "RemoveContainer" containerID="776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"
Nov 02 12:47:05.448215 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.448183    4928 scope.go:122] "RemoveContainer" containerID="776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"
Nov 02 12:47:05.448695 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:05.448641    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c\": container with ID starting with 776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c not found: ID does not exist" containerID="776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"
Nov 02 12:47:05.448798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.448689    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c"} err="failed to get container status \"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c\": rpc error: code = NotFound desc = could not find container \"776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c\": container with ID starting with 776a11d336e1298ade5529ffc661c54aad72a36f214fd2245bde4125d2f5c45c not found: ID does not exist"
Nov 02 12:47:05.489686 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.489647    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:47:05.492821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:05.492793    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0cf7e-fb9f4cd7-kwtk5"]
Nov 02 12:47:06.498547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:06.498503    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" path="/var/lib/kubelet/pods/7983f6ed-cfd2-43e1-85ec-284d6c6cd37e/volumes"
Nov 02 12:47:09.737040 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:09.736998    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:47:10.084378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:10.084298    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:47:14.886553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:14.886517    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:47:14.887023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:14.886712    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy" containerID="cri-o://2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d" gracePeriod=30
Nov 02 12:47:14.996530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:14.983715    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:47:14.996530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:14.984039    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy" containerID="cri-o://c9485fcd8f93bcf4fb9a6c23e9384cbf56bd4f623bd3e95df4330866e5520a88" gracePeriod=30
Nov 02 12:47:15.087852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.085304    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" podUID="542363cc-8598-4b64-950d-1b0fff4db6bb" containerName="payload-processing" containerID="cri-o://b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0" gracePeriod=25
Nov 02 12:47:15.472885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.472850    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:47:15.503295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.503260    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\" (UniqueName: \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\") pod \"542363cc-8598-4b64-950d-1b0fff4db6bb\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") "
Nov 02 12:47:15.503578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.503560    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\") pod \"542363cc-8598-4b64-950d-1b0fff4db6bb\" (UID: \"542363cc-8598-4b64-950d-1b0fff4db6bb\") "
Nov 02 12:47:15.504276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.504239    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume" pod "542363cc-8598-4b64-950d-1b0fff4db6bb" (UID: "542363cc-8598-4b64-950d-1b0fff4db6bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:15.506251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.506214    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv" pod "542363cc-8598-4b64-950d-1b0fff4db6bb" (UID: "542363cc-8598-4b64-950d-1b0fff4db6bb"). InnerVolumeSpecName "kube-api-access-4thsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:15.513604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.513567    4928 generic.go:356] "Generic (PLEG): container finished" podID="542363cc-8598-4b64-950d-1b0fff4db6bb" containerID="b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0" exitCode=0
Nov 02 12:47:15.513723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.513627    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"
Nov 02 12:47:15.513723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.513648    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" event={"ID":"542363cc-8598-4b64-950d-1b0fff4db6bb","Type":"ContainerDied","Data":"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"}
Nov 02 12:47:15.513848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.513733    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm" event={"ID":"542363cc-8598-4b64-950d-1b0fff4db6bb","Type":"ContainerDied","Data":"a171131270e64732ebb2dcb3adef04a5e8813ede31da0ee19639c488ce0169b1"}
Nov 02 12:47:15.513848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.513762    4928 scope.go:122] "RemoveContainer" containerID="b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"
Nov 02 12:47:15.536085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.535160    4928 scope.go:122] "RemoveContainer" containerID="b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"
Nov 02 12:47:15.536085 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:15.535536    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0\": container with ID starting with b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0 not found: ID does not exist" containerID="b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"
Nov 02 12:47:15.536085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.535572    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0"} err="failed to get container status \"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0\": rpc error: code = NotFound desc = could not find container \"b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0\": container with ID starting with b7a0c10930022aac1f23e06be007bf88c1d8958390e91c245652e49220845af0 not found: ID does not exist"
Nov 02 12:47:15.577077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.577037    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:47:15.585534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.585493    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-5968d699ff-z6pxm"]
Nov 02 12:47:15.605500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.605464    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thsv\" (UniqueName: \"kubernetes.io/projected/542363cc-8598-4b64-950d-1b0fff4db6bb-kube-api-access-4thsv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:15.605500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:15.605498    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542363cc-8598-4b64-950d-1b0fff4db6bb-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:16.500237 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:16.500200    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542363cc-8598-4b64-950d-1b0fff4db6bb" path="/var/lib/kubelet/pods/542363cc-8598-4b64-950d-1b0fff4db6bb/volumes"
Nov 02 12:47:20.406618 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.406250    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424623    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424665    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424740    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424809    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424896    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.424939    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.425108    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.425148    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.455850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.425251    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\") pod \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\" (UID: \"cd446b0c-2951-4da4-9d28-6cdfec83dbaa\") "
Nov 02 12:47:20.496635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.496228    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.503092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.500351    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:20.503783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.503738    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.504916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.504884    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.505042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.504962    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.505042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.505013    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.505719 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.505693    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "kube-api-access-l79zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.506113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.506080    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:20.507939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.507901    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy" pod "cd446b0c-2951-4da4-9d28-6cdfec83dbaa" (UID: "cd446b0c-2951-4da4-9d28-6cdfec83dbaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.597790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597687    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.597790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597737    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.597790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597754    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.597790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597773    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79zt\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-kube-api-access-l79zt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.597790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597791    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.598229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597808    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.598229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597836    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.598229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597848    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.598229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.597859    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd446b0c-2951-4da4-9d28-6cdfec83dbaa-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.697668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.696027    4928 generic.go:356] "Generic (PLEG): container finished" podID="18215d47-193b-42f6-9571-957b9b72e581" containerID="c9485fcd8f93bcf4fb9a6c23e9384cbf56bd4f623bd3e95df4330866e5520a88" exitCode=0
Nov 02 12:47:20.697668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.696180    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" event={"ID":"18215d47-193b-42f6-9571-957b9b72e581","Type":"ContainerDied","Data":"c9485fcd8f93bcf4fb9a6c23e9384cbf56bd4f623bd3e95df4330866e5520a88"}
Nov 02 12:47:20.700867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.700746    4928 generic.go:356] "Generic (PLEG): container finished" podID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerID="2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d" exitCode=0
Nov 02 12:47:20.700867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.700857    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"
Nov 02 12:47:20.701038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.700871    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" event={"ID":"cd446b0c-2951-4da4-9d28-6cdfec83dbaa","Type":"ContainerDied","Data":"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"}
Nov 02 12:47:20.701038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.700936    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc" event={"ID":"cd446b0c-2951-4da4-9d28-6cdfec83dbaa","Type":"ContainerDied","Data":"d3b52089722f13efd75721cc7619e362c4e10a6eaec1bdb237f283e7b091a742"}
Nov 02 12:47:20.701038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.700969    4928 scope.go:122] "RemoveContainer" containerID="2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"
Nov 02 12:47:20.719864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.719839    4928 scope.go:122] "RemoveContainer" containerID="2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"
Nov 02 12:47:20.720209 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:20.720180    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d\": container with ID starting with 2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d not found: ID does not exist" containerID="2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"
Nov 02 12:47:20.720281 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.720222    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d"} err="failed to get container status \"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d\": rpc error: code = NotFound desc = could not find container \"2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d\": container with ID starting with 2b5e44f1cbc4f0cde678498bdfbd81d8824bcca015bc2eda53db88e1057eb02d not found: ID does not exist"
Nov 02 12:47:20.812642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.812598    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:47:20.813905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.813873    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-jhwxc"]
Nov 02 12:47:20.905900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:20.905869    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:47:21.007335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007292    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007379    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007494    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007576    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007629    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007625    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:21.007861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007727    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007883    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.007936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007909    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:21.007936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007915    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.008090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.007984    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\") pod \"18215d47-193b-42f6-9571-957b9b72e581\" (UID: \"18215d47-193b-42f6-9571-957b9b72e581\") "
Nov 02 12:47:21.008532 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.008505    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:21.008884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.008861    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.008962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.008889    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.008962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.008904    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.009052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.009029    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:21.009269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.009244    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:21.011703 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.011670    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "kube-api-access-trpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:21.011877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.011845    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:21.011993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.011885    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:21.012458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.012431    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token" pod "18215d47-193b-42f6-9571-957b9b72e581" (UID: "18215d47-193b-42f6-9571-957b9b72e581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:21.110238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110200    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18215d47-193b-42f6-9571-957b9b72e581-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.110238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110237    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.110466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110249    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18215d47-193b-42f6-9571-957b9b72e581-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.110466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110278    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.110466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110292    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trpkt\" (UniqueName: \"kubernetes.io/projected/18215d47-193b-42f6-9571-957b9b72e581-kube-api-access-trpkt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.110466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.110304    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18215d47-193b-42f6-9571-957b9b72e581-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:21.710819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.710784    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"
Nov 02 12:47:21.711247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.710818    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h" event={"ID":"18215d47-193b-42f6-9571-957b9b72e581","Type":"ContainerDied","Data":"da7cd510dfd02e44c8960238064fb5d32afe59638bccf4600f7f2260c96e326e"}
Nov 02 12:47:21.711247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.710940    4928 scope.go:122] "RemoveContainer" containerID="c9485fcd8f93bcf4fb9a6c23e9384cbf56bd4f623bd3e95df4330866e5520a88"
Nov 02 12:47:21.815593 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.815555    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:47:21.861852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:21.860025    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-npj5h"]
Nov 02 12:47:22.515717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:22.515673    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18215d47-193b-42f6-9571-957b9b72e581" path="/var/lib/kubelet/pods/18215d47-193b-42f6-9571-957b9b72e581/volumes"
Nov 02 12:47:22.516276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:22.516254    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" path="/var/lib/kubelet/pods/cd446b0c-2951-4da4-9d28-6cdfec83dbaa/volumes"
Nov 02 12:47:27.991003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:27.990956    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:47:28.008316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:27.991061    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:47:28.852190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.852157    4928 generic.go:356] "Generic (PLEG): container finished" podID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerID="0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192" exitCode=0
Nov 02 12:47:28.852448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.852237    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerDied","Data":"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"}
Nov 02 12:47:28.852713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.852694    4928 scope.go:122] "RemoveContainer" containerID="0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"
Nov 02 12:47:28.854521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.854485    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:28.854672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.854571    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:28.854672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.854627    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:28.855029 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.854994    4928 generic.go:356] "Generic (PLEG): container finished" podID="95641b25-f013-40d4-bfae-35c996deea42" containerID="42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6" exitCode=0
Nov 02 12:47:28.855137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.855068    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerDied","Data":"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"}
Nov 02 12:47:28.855531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.855511    4928 scope.go:122] "RemoveContainer" containerID="42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"
Nov 02 12:47:28.857037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.857006    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:28.857146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.857088    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:28.857146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:28.857139    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:29.920692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:29.919197    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerStarted","Data":"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"}
Nov 02 12:47:29.967848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:29.925792    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerStarted","Data":"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"}
Nov 02 12:47:30.017662 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.017631    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:47:30.018579 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.018554    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:30.018645 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.018606    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:30.700872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.697739    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:47:30.700872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.698160    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:47:30.700872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:30.698213    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 02 12:47:31.017821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:31.017784    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:31.018242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:31.017864    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:31.697441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:31.697399    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:47:31.697693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:31.697670    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 02 12:47:31.867993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:31.867903    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:47:32.018170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:32.018133    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:32.018570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:32.018199    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:32.697640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:32.697602    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:47:32.697817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:32.697672    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 02 12:47:33.018154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:33.018124    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:33.018393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:33.018369    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:33.697056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:33.697017    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:47:33.697252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:33.697084    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 02 12:47:34.017723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:34.017686    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:34.017932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:34.017754    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:34.701354 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:34.701311    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:47:34.702073 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:34.702035    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 02 12:47:35.020596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:35.020561    4928 patch_prober.go:29] interesting pod/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 12:47:35.020915 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:35.020877    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 12:47:35.946984 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:35.946918    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:47:35.947807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:35.947779    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:47:35.948693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:35.948668    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:47:36.024043 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:36.024008    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:47:36.025226 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:36.025198    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:47:39.668553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:39.668520    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:47:39.706356 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:39.706324    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:47:39.732029 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:39.731986    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:47:43.203503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:43.203471    4928 scope.go:122] "RemoveContainer" containerID="d41671a1ed070358106f68d631bf94841278b75f62cefe1564181668b351e2f7"
Nov 02 12:47:44.669386 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:44.669340    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" podUID="b0ab9e0c-f754-4784-a1f3-848211e58191" containerName="payload-processing" containerID="cri-o://47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb" gracePeriod=25
Nov 02 12:47:44.707460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:44.707379    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" podUID="1c194de8-33b5-4698-bc0e-a80db65055db" containerName="payload-pre-processing" containerID="cri-o://3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4" gracePeriod=25
Nov 02 12:47:45.013844 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.013797    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:47:45.083474 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.083439    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:47:45.163210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.163172    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\") pod \"b0ab9e0c-f754-4784-a1f3-848211e58191\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") "
Nov 02 12:47:45.163415 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.163397    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\" (UniqueName: \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\") pod \"b0ab9e0c-f754-4784-a1f3-848211e58191\" (UID: \"b0ab9e0c-f754-4784-a1f3-848211e58191\") "
Nov 02 12:47:45.163700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.163667    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume" pod "b0ab9e0c-f754-4784-a1f3-848211e58191" (UID: "b0ab9e0c-f754-4784-a1f3-848211e58191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:45.164286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.164255    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab9e0c-f754-4784-a1f3-848211e58191-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.165756 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.165726    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74" pod "b0ab9e0c-f754-4784-a1f3-848211e58191" (UID: "b0ab9e0c-f754-4784-a1f3-848211e58191"). InnerVolumeSpecName "kube-api-access-65j74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:45.201241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.201211    4928 generic.go:356] "Generic (PLEG): container finished" podID="1c194de8-33b5-4698-bc0e-a80db65055db" containerID="3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4" exitCode=0
Nov 02 12:47:45.201409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.201261    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"
Nov 02 12:47:45.201409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.201300    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" event={"ID":"1c194de8-33b5-4698-bc0e-a80db65055db","Type":"ContainerDied","Data":"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"}
Nov 02 12:47:45.201409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.201362    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69" event={"ID":"1c194de8-33b5-4698-bc0e-a80db65055db","Type":"ContainerDied","Data":"02144e6875d85a2e38e7260f3f93e560c73aef99be0cd54c720a03bb2d687707"}
Nov 02 12:47:45.201409 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.201393    4928 scope.go:122] "RemoveContainer" containerID="3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"
Nov 02 12:47:45.202939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.202913    4928 generic.go:356] "Generic (PLEG): container finished" podID="b0ab9e0c-f754-4784-a1f3-848211e58191" containerID="47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb" exitCode=0
Nov 02 12:47:45.203065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.202955    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"
Nov 02 12:47:45.203065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.203003    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" event={"ID":"b0ab9e0c-f754-4784-a1f3-848211e58191","Type":"ContainerDied","Data":"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"}
Nov 02 12:47:45.203065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.203047    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt" event={"ID":"b0ab9e0c-f754-4784-a1f3-848211e58191","Type":"ContainerDied","Data":"3ea64c511881e7d50c820a93cb9b86ee80bcb853ac57681d86516ab68cb5828e"}
Nov 02 12:47:45.222732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.222705    4928 scope.go:122] "RemoveContainer" containerID="3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"
Nov 02 12:47:45.223080 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:45.223060    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4\": container with ID starting with 3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4 not found: ID does not exist" containerID="3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"
Nov 02 12:47:45.223133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.223089    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4"} err="failed to get container status \"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4\": rpc error: code = NotFound desc = could not find container \"3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4\": container with ID starting with 3ea92f3a81921c63f2bf78bf0af8c6a9403cfc1b3d0235203860176f02d35af4 not found: ID does not exist"
Nov 02 12:47:45.223133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.223107    4928 scope.go:122] "RemoveContainer" containerID="47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"
Nov 02 12:47:45.233359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.233338    4928 scope.go:122] "RemoveContainer" containerID="47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"
Nov 02 12:47:45.233702 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:47:45.233676    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb\": container with ID starting with 47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb not found: ID does not exist" containerID="47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"
Nov 02 12:47:45.233773 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.233710    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb"} err="failed to get container status \"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb\": rpc error: code = NotFound desc = could not find container \"47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb\": container with ID starting with 47e252a50d0b8bb0c2682e9f0d4e2b40741431c6098cd99ba6b05353604455fb not found: ID does not exist"
Nov 02 12:47:45.265278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.265249    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\" (UniqueName: \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\") pod \"1c194de8-33b5-4698-bc0e-a80db65055db\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") "
Nov 02 12:47:45.265468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.265454    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\") pod \"1c194de8-33b5-4698-bc0e-a80db65055db\" (UID: \"1c194de8-33b5-4698-bc0e-a80db65055db\") "
Nov 02 12:47:45.265875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.265852    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65j74\" (UniqueName: \"kubernetes.io/projected/b0ab9e0c-f754-4784-a1f3-848211e58191-kube-api-access-65j74\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.265875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.265856    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume" pod "1c194de8-33b5-4698-bc0e-a80db65055db" (UID: "1c194de8-33b5-4698-bc0e-a80db65055db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:45.267414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.267390    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l" pod "1c194de8-33b5-4698-bc0e-a80db65055db" (UID: "1c194de8-33b5-4698-bc0e-a80db65055db"). InnerVolumeSpecName "kube-api-access-5ts6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:45.277379 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.277349    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:47:45.281300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.281260    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-766c5858bc-22nnt"]
Nov 02 12:47:45.367807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.367764    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ts6l\" (UniqueName: \"kubernetes.io/projected/1c194de8-33b5-4698-bc0e-a80db65055db-kube-api-access-5ts6l\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.367807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.367800    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c194de8-33b5-4698-bc0e-a80db65055db-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.578668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.578633    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:47:45.585496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:45.585470    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-77854689d7-72s69"]
Nov 02 12:47:46.502459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:46.502417    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c194de8-33b5-4698-bc0e-a80db65055db" path="/var/lib/kubelet/pods/1c194de8-33b5-4698-bc0e-a80db65055db/volumes"
Nov 02 12:47:46.503001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:46.502961    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ab9e0c-f754-4784-a1f3-848211e58191" path="/var/lib/kubelet/pods/b0ab9e0c-f754-4784-a1f3-848211e58191/volumes"
Nov 02 12:47:55.565676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.565640    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:47:55.566598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566573    4928 cpu_manager.go:414] "containerMap: removing container" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy"
Nov 02 12:47:55.566598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566598    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566620    4928 cpu_manager.go:414] "containerMap: removing container" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566628    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566646    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" containerName="maas-api"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566653    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" containerName="maas-api"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566675    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1c194de8-33b5-4698-bc0e-a80db65055db" containerName="payload-pre-processing"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566683    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c194de8-33b5-4698-bc0e-a80db65055db" containerName="payload-pre-processing"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566704    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b0ab9e0c-f754-4784-a1f3-848211e58191" containerName="payload-processing"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566711    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ab9e0c-f754-4784-a1f3-848211e58191" containerName="payload-processing"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566723    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566729    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566739    4928 cpu_manager.go:414] "containerMap: removing container" podUID="542363cc-8598-4b64-950d-1b0fff4db6bb" containerName="payload-processing"
Nov 02 12:47:55.566781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566746    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542363cc-8598-4b64-950d-1b0fff4db6bb" containerName="payload-processing"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566966    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ab9e0c-f754-4784-a1f3-848211e58191" containerName="payload-processing"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566985    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58dba09-86a0-4bdd-b0a1-ef1431850452" containerName="istio-proxy"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.566995    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="542363cc-8598-4b64-950d-1b0fff4db6bb" containerName="payload-processing"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.567014    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="18215d47-193b-42f6-9571-957b9b72e581" containerName="istio-proxy"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.567026    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd446b0c-2951-4da4-9d28-6cdfec83dbaa" containerName="istio-proxy"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.567037    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7983f6ed-cfd2-43e1-85ec-284d6c6cd37e" containerName="maas-api"
Nov 02 12:47:55.567358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.567049    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c194de8-33b5-4698-bc0e-a80db65055db" containerName="payload-pre-processing"
Nov 02 12:47:55.572426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.572402    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.602227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.602194    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:47:55.610042 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.610004    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ee509d-openshift-default-dockercfg-hqdpv\""
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720112    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720177    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720290    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720323    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720493    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720577    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720611    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zbbq\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720667    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.741856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.720713    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.825222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.825118    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.826338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.826308    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.828768    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zbbq\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.828855    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.828902    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.828982    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.829021    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.829130    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.829162    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.827242    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.830178    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.830463    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.830722    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.837860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.831041    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.869470 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.869433    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.870172 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.870035    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.874635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.874601    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zbbq\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.885796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.885429    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\") pod \"e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") " pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:55.890683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:55.890606    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:56.272526 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:56.272489    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:47:56.272706 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:47:56.272682    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446327c9_7b83_4cd3_ad30_35672b28cd48.slice/crio-9845a8a4c8042441b72f668fd112a4c352794bb28d6e784a7a4a59fafa82bf3e WatchSource:0}: Error finding container 9845a8a4c8042441b72f668fd112a4c352794bb28d6e784a7a4a59fafa82bf3e: Status 404 returned error can't find the container with id 9845a8a4c8042441b72f668fd112a4c352794bb28d6e784a7a4a59fafa82bf3e
Nov 02 12:47:56.275500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:56.275460    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:56.275619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:56.275525    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:56.275619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:56.275553    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:47:56.318986 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:56.318950    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerStarted","Data":"9845a8a4c8042441b72f668fd112a4c352794bb28d6e784a7a4a59fafa82bf3e"}
Nov 02 12:47:57.329004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:57.328964    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerStarted","Data":"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"}
Nov 02 12:47:57.892058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:57.892011    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:57.898581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:57.898550    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:57.962488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:57.962410    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podStartSLOduration=2.962388251 podStartE2EDuration="2.962388251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:57.41163488 +0000 UTC m=+1275.060688839" watchObservedRunningTime="2026-11-02 12:47:57.962388251 +0000 UTC m=+1275.611442210"
Nov 02 12:47:58.340875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:58.338870    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:47:58.340875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:47:58.340462    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:48:07.412873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.412820    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:48:07.418510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.418482    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.440592 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.437708    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-299e74-openshift-default-dockercfg-8sp7v\""
Nov 02 12:48:07.465871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.458406    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:48:07.534702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.534667    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5d8\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.534902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.534712    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535012    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535130 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535115    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535188 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535139    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535257    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535407    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535488    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.535567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.535525    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.638759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.638714    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5d8\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.639414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.639384    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.640136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.640116    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.640712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.640693    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.641128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.641108    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.655027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.654997    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.655325 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.655302    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.655479 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.655362    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.655653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.655633    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.655803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.655786    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.656101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.656075    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.665942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.639995    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.665942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.641502    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.666514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.666486    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.667406 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.667334    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.687535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.687498    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5d8\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.688314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.688287    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.688997 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.688944    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\") pod \"e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") " pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:07.743118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:07.743083    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:08.132295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.132262    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:48:08.132867 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:48:08.132819    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f069f0_27c5_4a7d_b75f_0cd9f82d4a27.slice/crio-9aeaaacb45416edcbd33c69601061783cf56385f6e6193fb7d580ffad92eb962 WatchSource:0}: Error finding container 9aeaaacb45416edcbd33c69601061783cf56385f6e6193fb7d580ffad92eb962: Status 404 returned error can't find the container with id 9aeaaacb45416edcbd33c69601061783cf56385f6e6193fb7d580ffad92eb962
Nov 02 12:48:08.135867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.135797    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:08.135940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.135903    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:08.135940 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.135936    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:08.517936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.517898    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerStarted","Data":"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"}
Nov 02 12:48:08.518338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.517960    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerStarted","Data":"9aeaaacb45416edcbd33c69601061783cf56385f6e6193fb7d580ffad92eb962"}
Nov 02 12:48:08.583069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.583017    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podStartSLOduration=1.5830027599999998 podStartE2EDuration="1.58300276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:08.578594646 +0000 UTC m=+1286.227648604" watchObservedRunningTime="2026-11-02 12:48:08.58300276 +0000 UTC m=+1286.232056787"
Nov 02 12:48:08.746896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:08.744607    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:09.731819 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:09.731774    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:48:09.750424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:09.750400    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:10.537119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:10.537087    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:10.538104 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:10.538075    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:48:17.205338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:17.205297    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:48:17.205971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:17.205595    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy" containerID="cri-o://ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a" gracePeriod=30
Nov 02 12:48:17.218471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:17.218443    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:48:17.218658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:17.218636    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy" containerID="cri-o://5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625" gracePeriod=30
Nov 02 12:48:20.785239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:20.785202    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:48:20.786105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:20.786061    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" podUID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" containerName="maas-api" containerID="cri-o://128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1" gracePeriod=30
Nov 02 12:48:21.132732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.132703    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:48:21.185069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.185035    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\" (UniqueName: \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\") pod \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") "
Nov 02 12:48:21.185258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.185091    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\") pod \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") "
Nov 02 12:48:21.185258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.185134    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\") pod \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") "
Nov 02 12:48:21.185576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.185540    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca" pod "9c9881cd-5dac-4cbf-9e84-11133c0c3667" (UID: "9c9881cd-5dac-4cbf-9e84-11133c0c3667"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:21.186169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.186139    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\") pod \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\" (UID: \"9c9881cd-5dac-4cbf-9e84-11133c0c3667\") "
Nov 02 12:48:21.186925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.186898    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9881cd-5dac-4cbf-9e84-11133c0c3667-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.187686 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.187662    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q" pod "9c9881cd-5dac-4cbf-9e84-11133c0c3667" (UID: "9c9881cd-5dac-4cbf-9e84-11133c0c3667"). InnerVolumeSpecName "kube-api-access-7ck8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:21.187800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.187764    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs" pod "9c9881cd-5dac-4cbf-9e84-11133c0c3667" (UID: "9c9881cd-5dac-4cbf-9e84-11133c0c3667"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:21.188430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.188407    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls" pod "9c9881cd-5dac-4cbf-9e84-11133c0c3667" (UID: "9c9881cd-5dac-4cbf-9e84-11133c0c3667"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:21.288052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.288014    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ck8q\" (UniqueName: \"kubernetes.io/projected/9c9881cd-5dac-4cbf-9e84-11133c0c3667-kube-api-access-7ck8q\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.288052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.288056    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.288307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.288068    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c9881cd-5dac-4cbf-9e84-11133c0c3667-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.659970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.659937    4928 generic.go:356] "Generic (PLEG): container finished" podID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" containerID="128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1" exitCode=0
Nov 02 12:48:21.660166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.659979    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"
Nov 02 12:48:21.660166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.660012    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" event={"ID":"9c9881cd-5dac-4cbf-9e84-11133c0c3667","Type":"ContainerDied","Data":"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"}
Nov 02 12:48:21.660166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.660072    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j" event={"ID":"9c9881cd-5dac-4cbf-9e84-11133c0c3667","Type":"ContainerDied","Data":"943fe3e239dc47a311c457bfb4ae1ad5f303cfbc7915d83cf8261c16c9cd803e"}
Nov 02 12:48:21.660166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.660102    4928 scope.go:122] "RemoveContainer" containerID="128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"
Nov 02 12:48:21.672760 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.672735    4928 scope.go:122] "RemoveContainer" containerID="128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"
Nov 02 12:48:21.673059 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:21.673035    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1\": container with ID starting with 128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1 not found: ID does not exist" containerID="128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"
Nov 02 12:48:21.673124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.673071    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1"} err="failed to get container status \"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1\": rpc error: code = NotFound desc = could not find container \"128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1\": container with ID starting with 128a5d6b89b5046cc37712c8c6637dd08fd32e5ec1829e18be4760c44fd2c5a1 not found: ID does not exist"
Nov 02 12:48:21.726511 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.726461    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:48:21.739102 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:21.739032    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e6ceda-ddc85bb4d-d5t5j"]
Nov 02 12:48:22.510870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.510844    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" path="/var/lib/kubelet/pods/9c9881cd-5dac-4cbf-9e84-11133c0c3667/volumes"
Nov 02 12:48:22.535051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.535024    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:48:22.543307 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.543285    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:48:22.672482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.672440    4928 generic.go:356] "Generic (PLEG): container finished" podID="608a9672-4410-4436-84d8-4bbecfb33439" containerID="5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625" exitCode=0
Nov 02 12:48:22.672666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.672485    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"
Nov 02 12:48:22.672666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.672532    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" event={"ID":"608a9672-4410-4436-84d8-4bbecfb33439","Type":"ContainerDied","Data":"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"}
Nov 02 12:48:22.672666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.672601    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9" event={"ID":"608a9672-4410-4436-84d8-4bbecfb33439","Type":"ContainerDied","Data":"f3cac6d77712b8351d524294abcf8b7d312ace0697a6c17c0234a891ed9cfc07"}
Nov 02 12:48:22.672666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.672635    4928 scope.go:122] "RemoveContainer" containerID="5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"
Nov 02 12:48:22.676085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.676050    4928 generic.go:356] "Generic (PLEG): container finished" podID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerID="ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a" exitCode=0
Nov 02 12:48:22.676240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.676106    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"
Nov 02 12:48:22.676240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.676116    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" event={"ID":"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45","Type":"ContainerDied","Data":"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"}
Nov 02 12:48:22.676240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.676159    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j" event={"ID":"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45","Type":"ContainerDied","Data":"56fb5379ad1516cd0a1a8bae87b0ef2fd15ff9b9cd4c971fc907781255a3ece5"}
Nov 02 12:48:22.684336 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.684312    4928 scope.go:122] "RemoveContainer" containerID="5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"
Nov 02 12:48:22.684655 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:22.684618    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625\": container with ID starting with 5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625 not found: ID does not exist" containerID="5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"
Nov 02 12:48:22.684743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.684666    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625"} err="failed to get container status \"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625\": rpc error: code = NotFound desc = could not find container \"5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625\": container with ID starting with 5d70cff4e8a479bbca8b20f57447686021021ad616806cb10150e295e4637625 not found: ID does not exist"
Nov 02 12:48:22.684743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.684688    4928 scope.go:122] "RemoveContainer" containerID="ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"
Nov 02 12:48:22.698235 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.698210    4928 scope.go:122] "RemoveContainer" containerID="ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"
Nov 02 12:48:22.698639 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:22.698608    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a\": container with ID starting with ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a not found: ID does not exist" containerID="ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"
Nov 02 12:48:22.698746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.698648    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a"} err="failed to get container status \"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a\": rpc error: code = NotFound desc = could not find container \"ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a\": container with ID starting with ca1cd12cfa94d8f8b153c7562c002c7adc3f67b1bdfd36f4449197e7c988857a not found: ID does not exist"
Nov 02 12:48:22.712392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712369    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712417    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712446    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.712499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712496    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712517    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712546    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.712663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712614    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712623    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712695    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712759    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712783    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712854    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712875    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.712927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712901    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712959    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.712991    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713041    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713093    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713137    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713173    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.713238 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713202    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713264    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\") pod \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\" (UID: \"c4ffd476-1f2b-4ce8-a2e5-1991d040cf45\") "
Nov 02 12:48:22.713619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713288    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.713619 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713320    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\") pod \"608a9672-4410-4436-84d8-4bbecfb33439\" (UID: \"608a9672-4410-4436-84d8-4bbecfb33439\") "
Nov 02 12:48:22.713784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.713763    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.714125 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714095    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.714220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714201    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714224    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714225    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:22.714282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714240    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714266    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714283    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714297    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714313    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.714510 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.714374    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.715675 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.715647    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "kube-api-access-fjh2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:22.716347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716312    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:22.716443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716342    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:22.716443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716354    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:22.716443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716352    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.716443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716379    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928" pod "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" (UID: "c4ffd476-1f2b-4ce8-a2e5-1991d040cf45"). InnerVolumeSpecName "kube-api-access-gg928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:22.716443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716403    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:22.716788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.716769    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy" pod "608a9672-4410-4436-84d8-4bbecfb33439" (UID: "608a9672-4410-4436-84d8-4bbecfb33439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816224    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608a9672-4410-4436-84d8-4bbecfb33439-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816258    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608a9672-4410-4436-84d8-4bbecfb33439-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816269    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816278    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816287    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816295    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg928\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-kube-api-access-gg928\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816311    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816320    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816327    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816338    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjh2v\" (UniqueName: \"kubernetes.io/projected/608a9672-4410-4436-84d8-4bbecfb33439-kube-api-access-fjh2v\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:22.816688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:22.816346    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608a9672-4410-4436-84d8-4bbecfb33439-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:23.030817 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:23.030785    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:48:23.036658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:23.036628    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-z57f9"]
Nov 02 12:48:23.083587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:23.083514    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:48:23.091969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:23.091943    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-pvg7j"]
Nov 02 12:48:23.725508 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:23.725419    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:48:24.126492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.126454    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:48:24.127513 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127482    4928 cpu_manager.go:414] "containerMap: removing container" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy"
Nov 02 12:48:24.127513 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127512    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy"
Nov 02 12:48:24.127734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127529    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" containerName="maas-api"
Nov 02 12:48:24.127734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127538    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" containerName="maas-api"
Nov 02 12:48:24.127734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127567    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy"
Nov 02 12:48:24.127734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127613    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy"
Nov 02 12:48:24.127989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127854    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="608a9672-4410-4436-84d8-4bbecfb33439" containerName="istio-proxy"
Nov 02 12:48:24.127989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127870    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9881cd-5dac-4cbf-9e84-11133c0c3667" containerName="maas-api"
Nov 02 12:48:24.127989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.127884    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" containerName="istio-proxy"
Nov 02 12:48:24.133583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.133554    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.150256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.150231    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:48:24.157230 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.157202    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.157344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.157242    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.157344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.157263    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ee509d-serving-cert\""
Nov 02 12:48:24.157476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.157449    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56cp\" (UniqueName: \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.158184 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.157631    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.259340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.259300    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.259523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.259413    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.259523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.259436    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.259523 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:24.259449    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-serving-cert: secret "maas-api-e2e-praxis-a-ee509d-serving-cert" not found
Nov 02 12:48:24.259681 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:24.259522    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls podName:b634c570-9a63-4625-909c-55974c7af360 nodeName:}" failed. No retries permitted until 2026-11-02 12:48:24.75949969 +0000 UTC m=+1302.408553638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls") pod "maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" (UID: "b634c570-9a63-4625-909c-55974c7af360") : secret "maas-api-e2e-praxis-a-ee509d-serving-cert" not found
Nov 02 12:48:24.259681 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.259600    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56cp\" (UniqueName: \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.260275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.260253    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.262046 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.262026    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.284289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.284267    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56cp\" (UniqueName: \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.544850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.540098    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608a9672-4410-4436-84d8-4bbecfb33439" path="/var/lib/kubelet/pods/608a9672-4410-4436-84d8-4bbecfb33439/volumes"
Nov 02 12:48:24.544850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.540773    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ffd476-1f2b-4ce8-a2e5-1991d040cf45" path="/var/lib/kubelet/pods/c4ffd476-1f2b-4ce8-a2e5-1991d040cf45/volumes"
Nov 02 12:48:24.771407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.771373    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:24.773877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:24.773852    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:25.050834 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:25.050802    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:25.390786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:25.389799    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:48:25.391333 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:25.391301    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy" containerID="cri-o://663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163" gracePeriod=30
Nov 02 12:48:25.395116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:25.395094    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:48:25.709311 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:25.709222    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" event={"ID":"b634c570-9a63-4625-909c-55974c7af360","Type":"ContainerStarted","Data":"5c8efb9a1e0166275adc2401c1bfa3e2b8feb1d00919c5a3bb8306d014ff99ad"}
Nov 02 12:48:26.719239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:26.719191    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" event={"ID":"b634c570-9a63-4625-909c-55974c7af360","Type":"ContainerStarted","Data":"c47e36f8ddc44db01524879a01fc7ee9fee930f78672a77b2b15e49c52bd878f"}
Nov 02 12:48:26.719644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:26.719289    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:26.769658 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:26.769597    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" podStartSLOduration=2.445705707 podStartE2EDuration="2.769581972s" totalImagesPullingTime="323.876265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:26.767019436 +0000 UTC m=+1304.416073393" watchObservedRunningTime="2026-11-02 12:48:26.769581972 +0000 UTC m=+1304.418635931"
Nov 02 12:48:28.729849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:28.726556    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" podUID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" containerName="payload-pre-processing" containerID="cri-o://1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791" gracePeriod=25
Nov 02 12:48:29.037308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.037273    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:48:29.191945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.191902    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\") pod \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") "
Nov 02 12:48:29.192289 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.192264    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\" (UniqueName: \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\") pod \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\" (UID: \"34bfdb1a-a95a-460c-a669-91ad43ad96d3\") "
Nov 02 12:48:29.192469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.192442    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume" pod "34bfdb1a-a95a-460c-a669-91ad43ad96d3" (UID: "34bfdb1a-a95a-460c-a669-91ad43ad96d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:29.193504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.193464    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bfdb1a-a95a-460c-a669-91ad43ad96d3-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.201880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.195093    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr" pod "34bfdb1a-a95a-460c-a669-91ad43ad96d3" (UID: "34bfdb1a-a95a-460c-a669-91ad43ad96d3"). InnerVolumeSpecName "kube-api-access-qglzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:29.298896 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.298798    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglzr\" (UniqueName: \"kubernetes.io/projected/34bfdb1a-a95a-460c-a669-91ad43ad96d3-kube-api-access-qglzr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.747535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.747498    4928 generic.go:356] "Generic (PLEG): container finished" podID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" containerID="1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791" exitCode=0
Nov 02 12:48:29.748036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.747547    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"
Nov 02 12:48:29.748036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.747584    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" event={"ID":"34bfdb1a-a95a-460c-a669-91ad43ad96d3","Type":"ContainerDied","Data":"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"}
Nov 02 12:48:29.748036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.747649    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq" event={"ID":"34bfdb1a-a95a-460c-a669-91ad43ad96d3","Type":"ContainerDied","Data":"74bba68d9dce4745e8f693af9694f97776ea0100bd60ffe28f451085eae2f77f"}
Nov 02 12:48:29.748036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.747677    4928 scope.go:122] "RemoveContainer" containerID="1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"
Nov 02 12:48:29.760497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.760475    4928 scope.go:122] "RemoveContainer" containerID="1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"
Nov 02 12:48:29.760820 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:29.760801    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791\": container with ID starting with 1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791 not found: ID does not exist" containerID="1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"
Nov 02 12:48:29.760889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.760863    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791"} err="failed to get container status \"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791\": rpc error: code = NotFound desc = could not find container \"1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791\": container with ID starting with 1bab624a2d454d18c69269b516d73df8400b6ec1b920d13acc001e069c5b4791 not found: ID does not exist"
Nov 02 12:48:29.805276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.805245    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:48:29.809801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:29.809775    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2ec05-5f876cd47f-c5rzq"]
Nov 02 12:48:30.498322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.498288    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" path="/var/lib/kubelet/pods/34bfdb1a-a95a-460c-a669-91ad43ad96d3/volumes"
Nov 02 12:48:30.688751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.688726    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:48:30.760773 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.760736    4928 generic.go:356] "Generic (PLEG): container finished" podID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerID="663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163" exitCode=0
Nov 02 12:48:30.761466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.760815    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"
Nov 02 12:48:30.761466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.760871    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerDied","Data":"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"}
Nov 02 12:48:30.761466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.760966    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj" event={"ID":"fd1bbde6-b3a4-4403-bd15-f0e07ab10543","Type":"ContainerDied","Data":"6f33484bf1d25c9f8898b2227d0951b1e1472a6964a7366269f4a4e7c6895e2f"}
Nov 02 12:48:30.761466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.761017    4928 scope.go:122] "RemoveContainer" containerID="663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"
Nov 02 12:48:30.774034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.774010    4928 scope.go:122] "RemoveContainer" containerID="aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"
Nov 02 12:48:30.784465 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.784441    4928 scope.go:122] "RemoveContainer" containerID="663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"
Nov 02 12:48:30.784781 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:30.784755    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163\": container with ID starting with 663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163 not found: ID does not exist" containerID="663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"
Nov 02 12:48:30.784881 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.784794    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163"} err="failed to get container status \"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163\": rpc error: code = NotFound desc = could not find container \"663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163\": container with ID starting with 663e74cff5aa365fd6d80795148045646a23668a98b7d88fb50cd3f4ec9fd163 not found: ID does not exist"
Nov 02 12:48:30.784881 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.784818    4928 scope.go:122] "RemoveContainer" containerID="aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"
Nov 02 12:48:30.785143 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:30.785116    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585\": container with ID starting with aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585 not found: ID does not exist" containerID="aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"
Nov 02 12:48:30.785251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.785148    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585"} err="failed to get container status \"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585\": rpc error: code = NotFound desc = could not find container \"aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585\": container with ID starting with aa5dde12628f6a80acefcea8d357491db282083feabb4df736acf62ca176c585 not found: ID does not exist"
Nov 02 12:48:30.822800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.822770    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.822969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.822818    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.822969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.822868    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.822969 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.822903    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.822968    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823112    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823131    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:30.823308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823168    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823199    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823233    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\") pod \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\" (UID: \"fd1bbde6-b3a4-4403-bd15-f0e07ab10543\") "
Nov 02 12:48:30.823308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823166    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:30.823553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823512    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:30.823553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823532    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:30.823665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.823571    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:30.824118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.824093    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.824253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.824122    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.824253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.824137    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.824253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.824153    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.824253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.824168    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.825330 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.825300    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:30.825523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.825490    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:30.825596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.825527    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "kube-api-access-n5jhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:30.825925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.825899    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo" pod "fd1bbde6-b3a4-4403-bd15-f0e07ab10543" (UID: "fd1bbde6-b3a4-4403-bd15-f0e07ab10543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:30.926100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.926005    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.926100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.926034    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.926100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.926043    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:30.926100 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:30.926053    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5jhw\" (UniqueName: \"kubernetes.io/projected/fd1bbde6-b3a4-4403-bd15-f0e07ab10543-kube-api-access-n5jhw\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:48:31.116118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:31.116086    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:48:31.130862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:31.130799    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6ceda-openshift-default-56fffbb7-8qpgj"]
Nov 02 12:48:32.499534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:32.499497    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" path="/var/lib/kubelet/pods/fd1bbde6-b3a4-4403-bd15-f0e07ab10543/volumes"
Nov 02 12:48:32.732536 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:32.732506    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:48:35.515207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515174    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:48:35.515788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515775    4928 cpu_manager.go:414] "containerMap: removing container" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:48:35.515871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515790    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:48:35.515871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515803    4928 cpu_manager.go:414] "containerMap: removing container" podUID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" containerName="payload-pre-processing"
Nov 02 12:48:35.515871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515808    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" containerName="payload-pre-processing"
Nov 02 12:48:35.515871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515871    4928 cpu_manager.go:414] "containerMap: removing container" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:48:35.516016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515877    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:48:35.516016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.515987    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:48:35.516016 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.516001    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bfdb1a-a95a-460c-a669-91ad43ad96d3" containerName="payload-pre-processing"
Nov 02 12:48:35.519229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.519208    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.536889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.536857    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-299e74-serving-cert\""
Nov 02 12:48:35.571565 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.571523    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:48:35.597785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.597743    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.598149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.598118    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdr4\" (UniqueName: \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.598364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.598336    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.598459 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.598381    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.699524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.699485    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.699705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.699540    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdr4\" (UniqueName: \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.699705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.699578    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.699705 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.699594    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.700389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.700360    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.702098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.702066    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.702228 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.702141    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.739033 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.738998    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdr4\" (UniqueName: \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\") pod \"maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:35.838918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:35.836133    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:36.303082 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:36.303049    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:48:36.303598 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:48:36.303573    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ba0ed3_379c_49ec_843e_2c06a17f0ac3.slice/crio-5d7ca2443f8e1c7bcbec9264b40e93b6e2c59daf46f89030813f3c8a49df7739 WatchSource:0}: Error finding container 5d7ca2443f8e1c7bcbec9264b40e93b6e2c59daf46f89030813f3c8a49df7739: Status 404 returned error can't find the container with id 5d7ca2443f8e1c7bcbec9264b40e93b6e2c59daf46f89030813f3c8a49df7739
Nov 02 12:48:36.965973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:36.965869    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" event={"ID":"24ba0ed3-379c-49ec-843e-2c06a17f0ac3","Type":"ContainerStarted","Data":"4f9967b0e5cda8baed5ae8e448356f9f08ce09cba0f9471c098af5242a3b2663"}
Nov 02 12:48:36.965973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:36.965942    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" event={"ID":"24ba0ed3-379c-49ec-843e-2c06a17f0ac3","Type":"ContainerStarted","Data":"5d7ca2443f8e1c7bcbec9264b40e93b6e2c59daf46f89030813f3c8a49df7739"}
Nov 02 12:48:36.966382 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:36.965975    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:37.026067 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:37.016523    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" podStartSLOduration=1.661529676 podStartE2EDuration="2.016500393s" totalImagesPullingTime="354.970717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:37.014267897 +0000 UTC m=+1314.663321863" watchObservedRunningTime="2026-11-02 12:48:37.016500393 +0000 UTC m=+1314.665554397"
Nov 02 12:48:39.762130 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:48:39.762082    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:48:43.012851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:43.012157    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:48:44.381003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:44.380961    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:48:44.381417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:44.381046    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:48:47.086418 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.086383    4928 generic.go:356] "Generic (PLEG): container finished" podID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerID="80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175" exitCode=0
Nov 02 12:48:47.086890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.086453    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerDied","Data":"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"}
Nov 02 12:48:47.086944 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.086890    4928 scope.go:122] "RemoveContainer" containerID="80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"
Nov 02 12:48:47.088471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.088439    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:47.088571 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.088519    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:47.088571 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:47.088567    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:48:48.099482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:48.099439    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerStarted","Data":"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"}
Nov 02 12:48:48.892265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:48.892230    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:48:48.892544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:48.892435    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 12:48:48.892544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:48.892498    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 12:48:49.892260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:49.892222    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 12:48:49.892752 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:49.892278    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 12:48:50.892481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:50.892445    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 12:48:50.893010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:50.892504    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 12:48:51.892514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:51.892476    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 12:48:51.893052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:51.892531    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 12:48:52.892580 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:52.892541    4928 patch_prober.go:29] interesting pod/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 12:48:52.893008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:52.892613    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 12:48:53.897795 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:53.897762    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:48:53.898298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:53.898273    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:48:53.898931 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:48:53.898910    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:49:00.251613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:00.251572    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:49:00.252150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:00.251902    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy" containerID="cri-o://2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10" gracePeriod=30
Nov 02 12:49:05.607065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.607036    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:49:05.622366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622338    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622400    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622465    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622492    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622579    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622588    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:05.622692 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622611    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622690    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622727    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.622899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.622778    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\") pod \"95641b25-f013-40d4-bfae-35c996deea42\" (UID: \"95641b25-f013-40d4-bfae-35c996deea42\") "
Nov 02 12:49:05.623142 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623115    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:05.623269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623167    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:05.623337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623314    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:05.623403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623345    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:05.623561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623537    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95641b25-f013-40d4-bfae-35c996deea42-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.623629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623565    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.623629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623579    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.623629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623593    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.623629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.623607    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.635849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.633589    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:05.651847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.648929    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:49:05.651847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.649059    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:05.662210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.661490    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79" pod "95641b25-f013-40d4-bfae-35c996deea42" (UID: "95641b25-f013-40d4-bfae-35c996deea42"). InnerVolumeSpecName "kube-api-access-czq79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:05.756859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.725745    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95641b25-f013-40d4-bfae-35c996deea42-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.756859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.725778    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95641b25-f013-40d4-bfae-35c996deea42-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.756859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.725791    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:05.756859 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:05.725805    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czq79\" (UniqueName: \"kubernetes.io/projected/95641b25-f013-40d4-bfae-35c996deea42-kube-api-access-czq79\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:49:06.378907 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.378866    4928 generic.go:356] "Generic (PLEG): container finished" podID="95641b25-f013-40d4-bfae-35c996deea42" containerID="2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10" exitCode=0
Nov 02 12:49:06.379133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.378929    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"
Nov 02 12:49:06.379133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.378967    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerDied","Data":"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"}
Nov 02 12:49:06.379133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.379049    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw" event={"ID":"95641b25-f013-40d4-bfae-35c996deea42","Type":"ContainerDied","Data":"be58e8e34003fdaaadc8016cec53e06fe24a23087ace00a520df2a3571d2bbdf"}
Nov 02 12:49:06.379133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.379070    4928 scope.go:122] "RemoveContainer" containerID="2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"
Nov 02 12:49:06.391383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.391284    4928 scope.go:122] "RemoveContainer" containerID="42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"
Nov 02 12:49:06.401851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.401800    4928 scope.go:122] "RemoveContainer" containerID="2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"
Nov 02 12:49:06.402112 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:06.402093    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10\": container with ID starting with 2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10 not found: ID does not exist" containerID="2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"
Nov 02 12:49:06.402165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.402123    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10"} err="failed to get container status \"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10\": rpc error: code = NotFound desc = could not find container \"2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10\": container with ID starting with 2c09fa03605780d4abe72c505c156198c0f246e95bae57b243cf40c936e5cd10 not found: ID does not exist"
Nov 02 12:49:06.402165 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.402143    4928 scope.go:122] "RemoveContainer" containerID="42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"
Nov 02 12:49:06.402368 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:06.402349    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6\": container with ID starting with 42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6 not found: ID does not exist" containerID="42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"
Nov 02 12:49:06.402415 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.402374    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6"} err="failed to get container status \"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6\": rpc error: code = NotFound desc = could not find container \"42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6\": container with ID starting with 42d62a5ed123e3d981d45faf7bfecdcecc7ca0b2183669c913cf6fcb8d7c09d6 not found: ID does not exist"
Nov 02 12:49:06.466648 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.466613    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:49:06.471493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.471459    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-078fd3-openshift-default-d66545889-fccnw"]
Nov 02 12:49:06.499599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:06.499568    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95641b25-f013-40d4-bfae-35c996deea42" path="/var/lib/kubelet/pods/95641b25-f013-40d4-bfae-35c996deea42/volumes"
Nov 02 12:49:09.734114 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:09.734033    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:49:11.581993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:11.581951    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:49:11.582552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:11.582029    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:49:13.442374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.442328    4928 generic.go:356] "Generic (PLEG): container finished" podID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerID="883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31" exitCode=0
Nov 02 12:49:13.442980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.442406    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerDied","Data":"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"}
Nov 02 12:49:13.443066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.442981    4928 scope.go:122] "RemoveContainer" containerID="883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"
Nov 02 12:49:13.444587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.444553    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:49:13.444714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.444639    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:49:13.444714 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:13.444691    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:49:14.472774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:14.472736    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerStarted","Data":"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"}
Nov 02 12:49:14.748847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:14.745956    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:49:14.748847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:14.746092    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 12:49:14.748847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:14.746152    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 12:49:15.752852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:15.752599    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 12:49:15.752852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:15.752660    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 12:49:16.748848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:16.746227    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 12:49:16.748848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:16.746308    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 12:49:17.752864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:17.752809    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 12:49:17.753497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:17.753447    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 12:49:18.748043 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:18.747986    4928 patch_prober.go:29] interesting pod/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 12:49:18.748476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:18.748440    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 12:49:19.748861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:19.748797    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:49:19.749314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:19.749155    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:49:19.749939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:19.749912    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:49:39.300389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.300353    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:49:39.301257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301235    4928 cpu_manager.go:414] "containerMap: removing container" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.301257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301258    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.301416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301270    4928 cpu_manager.go:414] "containerMap: removing container" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.301416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301277    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.301416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301372    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1bbde6-b3a4-4403-bd15-f0e07ab10543" containerName="istio-proxy"
Nov 02 12:49:39.301416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301386    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.301416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.301393    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="95641b25-f013-40d4-bfae-35c996deea42" containerName="istio-proxy"
Nov 02 12:49:39.305398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.305377    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.348572 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.348532    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtd7m\" (UniqueName: \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.348757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.348591    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.348757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.348645    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.348757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.348708    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.350058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.350009    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eb24df-serving-cert\""
Nov 02 12:49:39.354598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.354567    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:49:39.450285 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.450247    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.450487 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.450347    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.450487 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:39.450472    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-serving-cert: secret "maas-api-e2e-ipp-a-eb24df-serving-cert" not found
Nov 02 12:49:39.450617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.450519    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtd7m\" (UniqueName: \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.450617 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:39.450560    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls podName:5c2f20db-ed9a-4be6-a81b-56d409c95db1 nodeName:}" failed. No retries permitted until 2026-11-02 12:49:39.950537639 +0000 UTC m=+1377.599591590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls") pod "maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" (UID: "5c2f20db-ed9a-4be6-a81b-56d409c95db1") : secret "maas-api-e2e-ipp-a-eb24df-serving-cert" not found
Nov 02 12:49:39.450617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.450608    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.451037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.451017    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.453174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.453150    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.474929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.474900    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtd7m\" (UniqueName: \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.731924 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:49:39.731820    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:49:39.958299 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.958264    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:39.960871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:39.960842    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:40.227695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:40.227644    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:40.584685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:40.584649    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:49:40.632016 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:49:40.631983    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2f20db_ed9a_4be6_a81b_56d409c95db1.slice/crio-d5c8a0206087d3be719fe4036197d6412e1a0d4e00ccd9156e283398e5f6e1b6 WatchSource:0}: Error finding container d5c8a0206087d3be719fe4036197d6412e1a0d4e00ccd9156e283398e5f6e1b6: Status 404 returned error can't find the container with id d5c8a0206087d3be719fe4036197d6412e1a0d4e00ccd9156e283398e5f6e1b6
Nov 02 12:49:40.740469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:40.740431    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" event={"ID":"5c2f20db-ed9a-4be6-a81b-56d409c95db1","Type":"ContainerStarted","Data":"d5c8a0206087d3be719fe4036197d6412e1a0d4e00ccd9156e283398e5f6e1b6"}
Nov 02 12:49:41.750819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:41.750774    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" event={"ID":"5c2f20db-ed9a-4be6-a81b-56d409c95db1","Type":"ContainerStarted","Data":"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"}
Nov 02 12:49:41.751417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:41.750901    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:41.802974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:41.802916    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" podStartSLOduration=2.532032828 podStartE2EDuration="2.802900215s" totalImagesPullingTime="270.867387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:41.801029978 +0000 UTC m=+1379.450083935" watchObservedRunningTime="2026-11-02 12:49:41.802900215 +0000 UTC m=+1379.451954187"
Nov 02 12:49:47.766083 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:47.766050    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:49:59.246247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.245955    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:49:59.253074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.253047    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.297265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.293739    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:49:59.303069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.303039    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d3a281-openshift-default-dockercfg-bbvz9\""
Nov 02 12:49:59.387461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387191    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387313    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387352    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387492    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387615    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387646    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387671    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387700    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxvf\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.387805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.387729    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495730    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495783    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495807    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495849    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxvf\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495879    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.495954    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.496042    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.496072    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.499899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.496193    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.503864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.501289    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.503864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.501705    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.503864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.502015    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.503864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.502135    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.503864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.502272    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.556723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.556651    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.560361 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.560315    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.560563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.560510    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxvf\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.568092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.567306    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\") pod \"e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") " pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.592296 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.592258    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:49:59.949756 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.949719    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:49:59.950757 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:49:59.950724    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbeeb8c_60ab_46ac_b868_b4c08f2be000.slice/crio-83c6396b19e962a0e745a99c94ec5290abad7f61bf59b3d53d6ba996b7fb57ba WatchSource:0}: Error finding container 83c6396b19e962a0e745a99c94ec5290abad7f61bf59b3d53d6ba996b7fb57ba: Status 404 returned error can't find the container with id 83c6396b19e962a0e745a99c94ec5290abad7f61bf59b3d53d6ba996b7fb57ba
Nov 02 12:49:59.953592 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.953540    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:49:59.953715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.953639    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:49:59.953782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:49:59.953709    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:50:00.916836 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:00.916785    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerStarted","Data":"2c530cc34df9ee239389872c007f062c31d609b6a0cb7574808e1a1672cff89a"}
Nov 02 12:50:00.917305 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:00.916853    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerStarted","Data":"83c6396b19e962a0e745a99c94ec5290abad7f61bf59b3d53d6ba996b7fb57ba"}
Nov 02 12:50:00.993300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:00.993240    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podStartSLOduration=1.993222891 podStartE2EDuration="1.993222891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:00.979068544 +0000 UTC m=+1398.628122503" watchObservedRunningTime="2026-11-02 12:50:00.993222891 +0000 UTC m=+1398.642276848"
Nov 02 12:50:01.592767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:01.592731    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:50:01.598906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:01.598880    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:50:01.924322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:01.924233    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:50:01.925348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:01.925324    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:50:09.730761 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:09.730707    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:50:22.887230 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:22.887179    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:50:22.887875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:22.887421    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" podUID="b634c570-9a63-4625-909c-55974c7af360" containerName="maas-api" containerID="cri-o://c47e36f8ddc44db01524879a01fc7ee9fee930f78672a77b2b15e49c52bd878f" gracePeriod=30
Nov 02 12:50:23.320623 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.320422    4928 generic.go:356] "Generic (PLEG): container finished" podID="b634c570-9a63-4625-909c-55974c7af360" containerID="c47e36f8ddc44db01524879a01fc7ee9fee930f78672a77b2b15e49c52bd878f" exitCode=0
Nov 02 12:50:23.320708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.320653    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" event={"ID":"b634c570-9a63-4625-909c-55974c7af360","Type":"ContainerDied","Data":"c47e36f8ddc44db01524879a01fc7ee9fee930f78672a77b2b15e49c52bd878f"}
Nov 02 12:50:23.410335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.410286    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:50:23.457809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.457773    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\") pod \"b634c570-9a63-4625-909c-55974c7af360\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") "
Nov 02 12:50:23.458845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.458803    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\" (UniqueName: \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\") pod \"b634c570-9a63-4625-909c-55974c7af360\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") "
Nov 02 12:50:23.459396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.459375    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") pod \"b634c570-9a63-4625-909c-55974c7af360\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") "
Nov 02 12:50:23.460004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.458473    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca" pod "b634c570-9a63-4625-909c-55974c7af360" (UID: "b634c570-9a63-4625-909c-55974c7af360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:23.460463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.460443    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\") pod \"b634c570-9a63-4625-909c-55974c7af360\" (UID: \"b634c570-9a63-4625-909c-55974c7af360\") "
Nov 02 12:50:23.462713 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.462694    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b634c570-9a63-4625-909c-55974c7af360-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:23.467787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.467677    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls" pod "b634c570-9a63-4625-909c-55974c7af360" (UID: "b634c570-9a63-4625-909c-55974c7af360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:23.468333 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.468311    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp" pod "b634c570-9a63-4625-909c-55974c7af360" (UID: "b634c570-9a63-4625-909c-55974c7af360"). InnerVolumeSpecName "kube-api-access-g56cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:23.468777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.468756    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs" pod "b634c570-9a63-4625-909c-55974c7af360" (UID: "b634c570-9a63-4625-909c-55974c7af360"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:23.564198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.564114    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:23.564198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.564147    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56cp\" (UniqueName: \"kubernetes.io/projected/b634c570-9a63-4625-909c-55974c7af360-kube-api-access-g56cp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:23.564198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:23.564161    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b634c570-9a63-4625-909c-55974c7af360-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:24.330024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.329989    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"
Nov 02 12:50:24.330024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.330016    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq" event={"ID":"b634c570-9a63-4625-909c-55974c7af360","Type":"ContainerDied","Data":"5c8efb9a1e0166275adc2401c1bfa3e2b8feb1d00919c5a3bb8306d014ff99ad"}
Nov 02 12:50:24.330588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.330092    4928 scope.go:122] "RemoveContainer" containerID="c47e36f8ddc44db01524879a01fc7ee9fee930f78672a77b2b15e49c52bd878f"
Nov 02 12:50:24.387819 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.387783    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:50:24.400476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.400446    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ee509d-7b69c4b84d-8ndlq"]
Nov 02 12:50:24.499353 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:24.499321    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b634c570-9a63-4625-909c-55974c7af360" path="/var/lib/kubelet/pods/b634c570-9a63-4625-909c-55974c7af360/volumes"
Nov 02 12:50:26.664210 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:26.664167    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:50:26.664694 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:26.664461    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy" containerID="cri-o://953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c" gracePeriod=30
Nov 02 12:50:29.715068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:29.715030    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:50:29.715580 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:29.715308    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy" containerID="cri-o://c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978" gracePeriod=30
Nov 02 12:50:32.070277 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.070245    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:50:32.245426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245377    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245476    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245510    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245546    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245653 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245623    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245667    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245704    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245706    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245748    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245813    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\") pod \"446327c9-7b83-4cd3-ad30-35672b28cd48\" (UID: \"446327c9-7b83-4cd3-ad30-35672b28cd48\") "
Nov 02 12:50:32.245916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.245880    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:32.246222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246051    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:32.246278 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246259    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:32.246637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246610    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:32.246741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246715    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.246800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246748    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.246800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246764    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.246800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.246778    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.248539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.248501    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:32.248668 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.248608    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:32.248727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.248687    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "kube-api-access-4zbbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:32.248865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.248846    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy" pod "446327c9-7b83-4cd3-ad30-35672b28cd48" (UID: "446327c9-7b83-4cd3-ad30-35672b28cd48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:32.347672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.347636    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.347672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.347669    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.347672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.347677    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446327c9-7b83-4cd3-ad30-35672b28cd48-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.347949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.347686    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zbbq\" (UniqueName: \"kubernetes.io/projected/446327c9-7b83-4cd3-ad30-35672b28cd48-kube-api-access-4zbbq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.347949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.347697    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446327c9-7b83-4cd3-ad30-35672b28cd48-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:32.397700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.397667    4928 generic.go:356] "Generic (PLEG): container finished" podID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerID="953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c" exitCode=0
Nov 02 12:50:32.397893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.397717    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"
Nov 02 12:50:32.397893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.397749    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerDied","Data":"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"}
Nov 02 12:50:32.397893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.397810    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt" event={"ID":"446327c9-7b83-4cd3-ad30-35672b28cd48","Type":"ContainerDied","Data":"9845a8a4c8042441b72f668fd112a4c352794bb28d6e784a7a4a59fafa82bf3e"}
Nov 02 12:50:32.397893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.397862    4928 scope.go:122] "RemoveContainer" containerID="953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"
Nov 02 12:50:32.409492 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.409472    4928 scope.go:122] "RemoveContainer" containerID="80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"
Nov 02 12:50:32.419820 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.419798    4928 scope.go:122] "RemoveContainer" containerID="953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"
Nov 02 12:50:32.420150 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:32.420133    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c\": container with ID starting with 953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c not found: ID does not exist" containerID="953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"
Nov 02 12:50:32.420204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.420160    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c"} err="failed to get container status \"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c\": rpc error: code = NotFound desc = could not find container \"953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c\": container with ID starting with 953f8e46b4512412914a36e66dbafdc02184ae98c0d0ced3ee5cb07ee183494c not found: ID does not exist"
Nov 02 12:50:32.420204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.420177    4928 scope.go:122] "RemoveContainer" containerID="80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"
Nov 02 12:50:32.420437 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:32.420420    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175\": container with ID starting with 80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175 not found: ID does not exist" containerID="80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"
Nov 02 12:50:32.420499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.420440    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175"} err="failed to get container status \"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175\": rpc error: code = NotFound desc = could not find container \"80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175\": container with ID starting with 80d3372588c9e5d71018b1dc5ef757dbaf4d5cae94794cf75e810e96adf27175 not found: ID does not exist"
Nov 02 12:50:32.459057 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.459022    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:50:32.484694 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.484654    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ee509d-openshift-default-5d7f44c765-twzdt"]
Nov 02 12:50:32.506204 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:32.506121    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" path="/var/lib/kubelet/pods/446327c9-7b83-4cd3-ad30-35672b28cd48/volumes"
Nov 02 12:50:35.017106 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.017077    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:50:35.035628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.035592    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.035842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.035647    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.035842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.035669    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.035842 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.035806    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.035940    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:35.036084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036014    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:35.036084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036020    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036146    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:35.036191 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036166    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036206    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:35.036304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036224    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036262    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036297    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\") pod \"1e3443c7-8492-4c37-8927-3e2315d3af30\" (UID: \"1e3443c7-8492-4c37-8927-3e2315d3af30\") "
Nov 02 12:50:35.036594 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.036569    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:35.037066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.037039    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.037187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.037071    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.037187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.037090    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.037187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.037103    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.037187 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.037117    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3443c7-8492-4c37-8927-3e2315d3af30-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.038957 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.038718    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "kube-api-access-mpjht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:35.038957 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.038869    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:35.039413 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.039380    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:35.040755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.040719    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo" pod "1e3443c7-8492-4c37-8927-3e2315d3af30" (UID: "1e3443c7-8492-4c37-8927-3e2315d3af30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:35.139196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.139158    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjht\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-kube-api-access-mpjht\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.139196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.139196    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.139430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.139209    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.139430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.139222    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3443c7-8492-4c37-8927-3e2315d3af30-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:35.548126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.548086    4928 generic.go:356] "Generic (PLEG): container finished" podID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerID="c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978" exitCode=0
Nov 02 12:50:35.548298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.548178    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"
Nov 02 12:50:35.548298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.548251    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerDied","Data":"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"}
Nov 02 12:50:35.548448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.548298    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz" event={"ID":"1e3443c7-8492-4c37-8927-3e2315d3af30","Type":"ContainerDied","Data":"4bf22485e6f41195bce664945edb8472ee25f7392b8c1794faacd882c7c69ff8"}
Nov 02 12:50:35.548448 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.548328    4928 scope.go:122] "RemoveContainer" containerID="c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"
Nov 02 12:50:35.561061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.561038    4928 scope.go:122] "RemoveContainer" containerID="1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"
Nov 02 12:50:35.572035 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.572007    4928 scope.go:122] "RemoveContainer" containerID="c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"
Nov 02 12:50:35.572317 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:35.572297    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978\": container with ID starting with c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978 not found: ID does not exist" containerID="c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"
Nov 02 12:50:35.572383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.572326    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978"} err="failed to get container status \"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978\": rpc error: code = NotFound desc = could not find container \"c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978\": container with ID starting with c67ddb9a4ac454262b1b0255163e87dd20d996289d25707ef59e72cb59a27978 not found: ID does not exist"
Nov 02 12:50:35.572383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.572346    4928 scope.go:122] "RemoveContainer" containerID="1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"
Nov 02 12:50:35.572640 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:35.572622    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1\": container with ID starting with 1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1 not found: ID does not exist" containerID="1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"
Nov 02 12:50:35.572754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.572645    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1"} err="failed to get container status \"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1\": rpc error: code = NotFound desc = could not find container \"1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1\": container with ID starting with 1adc15d0d619325b92a3a0f720a4f1982718185863d874fde25385ef44494ab1 not found: ID does not exist"
Nov 02 12:50:35.620614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.620568    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:50:35.634110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:35.634078    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-g4lqz"]
Nov 02 12:50:36.503264 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:36.503226    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" path="/var/lib/kubelet/pods/1e3443c7-8492-4c37-8927-3e2315d3af30/volumes"
Nov 02 12:50:39.731315 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:39.731270    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:50:43.258163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258122    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:50:43.258725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258711    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b634c570-9a63-4625-909c-55974c7af360" containerName="maas-api"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258726    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b634c570-9a63-4625-909c-55974c7af360" containerName="maas-api"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258747    4928 cpu_manager.go:414] "containerMap: removing container" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258753    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258766    4928 cpu_manager.go:414] "containerMap: removing container" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258771    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258788    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.258793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258793    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258806    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258810    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258921    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258931    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3443c7-8492-4c37-8927-3e2315d3af30" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258941    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b634c570-9a63-4625-909c-55974c7af360" containerName="maas-api"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258950    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.259044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.258956    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="446327c9-7b83-4cd3-ad30-35672b28cd48" containerName="istio-proxy"
Nov 02 12:50:43.264124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.264100    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.284040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.284010    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9750c7-openshift-default-dockercfg-6f96j\""
Nov 02 12:50:43.300293 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.300256    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:50:43.385369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385335    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385373    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385417    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385491    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385613    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385700 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385674    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385719    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zq46\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385855    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.385922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.385893    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.487628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.487584    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.487849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.487658    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.487849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.487708    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq46\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.487943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.487897    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.487989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.487955    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.488058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488031    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.488201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488068    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.488201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488140    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.488201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488167    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.488726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488701    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.489021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.488989    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.489119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.489036    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.490352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.490057    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.490352 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.490173    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.491194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.491167    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.492358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.492336    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.513771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.513703    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.513956 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.513864    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zq46\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.578358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.578326    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:43.929094 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:50:43.929060    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b768ba_8fae_4d31_af22_7bdd7aab289c.slice/crio-965825c6dfa3757483d6733d3eb70c169e2e69222f5c419edae2ed436d7fd9e3 WatchSource:0}: Error finding container 965825c6dfa3757483d6733d3eb70c169e2e69222f5c419edae2ed436d7fd9e3: Status 404 returned error can't find the container with id 965825c6dfa3757483d6733d3eb70c169e2e69222f5c419edae2ed436d7fd9e3
Nov 02 12:50:43.931733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.931694    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:50:43.931865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.931764    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:50:43.931865 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.931795    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:50:43.932193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:43.932160    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:50:44.671084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:44.671045    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" event={"ID":"d8b768ba-8fae-4d31-af22-7bdd7aab289c","Type":"ContainerStarted","Data":"699e19b38b0fa53fc9027304ef31d52584e6fabf215e2ebb56d15bc4a4c3d90b"}
Nov 02 12:50:44.671590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:44.671112    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" event={"ID":"d8b768ba-8fae-4d31-af22-7bdd7aab289c","Type":"ContainerStarted","Data":"965825c6dfa3757483d6733d3eb70c169e2e69222f5c419edae2ed436d7fd9e3"}
Nov 02 12:50:44.756467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:44.756407    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podStartSLOduration=1.756389258 podStartE2EDuration="1.756389258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:44.740321309 +0000 UTC m=+1442.389375271" watchObservedRunningTime="2026-11-02 12:50:44.756389258 +0000 UTC m=+1442.405443215"
Nov 02 12:50:45.578520 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:45.578486    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:45.580092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:45.580064    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 12:50:45.580227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:45.580122    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 12:50:46.592851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:46.588500    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 12:50:46.592851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:46.588582    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 12:50:47.579678 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:47.579638    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 12:50:47.579980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:47.579694    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 12:50:48.579603 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:48.579567    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 12:50:48.580061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:48.579637    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 12:50:49.578901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:49.578865    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 02 12:50:49.579087 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:49.578937    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 02 12:50:50.583588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:50.583556    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:50.584093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:50.584071    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:50.584667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:50.584645    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:50:51.597601 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:51.597566    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:50:51.598163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:51.597781    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy" containerID="cri-o://08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422" gracePeriod=30
Nov 02 12:50:56.893634 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.893601    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:50:56.910167 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.910127    4928 generic.go:356] "Generic (PLEG): container finished" podID="7fe4fa08-5563-4674-976c-de81a3a58148" containerID="08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422" exitCode=0
Nov 02 12:50:56.910345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.910177    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"
Nov 02 12:50:56.910345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.910213    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" event={"ID":"7fe4fa08-5563-4674-976c-de81a3a58148","Type":"ContainerDied","Data":"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"}
Nov 02 12:50:56.910345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.910269    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr" event={"ID":"7fe4fa08-5563-4674-976c-de81a3a58148","Type":"ContainerDied","Data":"b007b4e26478eea6897cf2875a58c200ce3453836915dff1373c81b896bb77bf"}
Nov 02 12:50:56.910345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.910296    4928 scope.go:122] "RemoveContainer" containerID="08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"
Nov 02 12:50:56.921593 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.921572    4928 scope.go:122] "RemoveContainer" containerID="08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"
Nov 02 12:50:56.921924 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:50:56.921902    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422\": container with ID starting with 08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422 not found: ID does not exist" containerID="08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"
Nov 02 12:50:56.921989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:56.921936    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422"} err="failed to get container status \"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422\": rpc error: code = NotFound desc = could not find container \"08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422\": container with ID starting with 08c70cb152a47c802cf51f5780b3a0da401bb08faa919fd47ea9ded0afdfc422 not found: ID does not exist"
Nov 02 12:50:57.037901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.037861    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.037929    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.037997    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038132    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038101    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:57.038218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038187    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038246    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038310    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038373 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038350    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038525 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038419    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\") pod \"7fe4fa08-5563-4674-976c-de81a3a58148\" (UID: \"7fe4fa08-5563-4674-976c-de81a3a58148\") "
Nov 02 12:50:57.038576 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038541    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.038738    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.039122    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.039175    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.039207    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.039229    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.039315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.039243    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.041040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.040986    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:57.041040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.041026    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "kube-api-access-8dtcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:57.041371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.041340    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:57.041516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.041497    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo" pod "7fe4fa08-5563-4674-976c-de81a3a58148" (UID: "7fe4fa08-5563-4674-976c-de81a3a58148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141060    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141096    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe4fa08-5563-4674-976c-de81a3a58148-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141110    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe4fa08-5563-4674-976c-de81a3a58148-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141123    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe4fa08-5563-4674-976c-de81a3a58148-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141137    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.141150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.141151    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtcx\" (UniqueName: \"kubernetes.io/projected/7fe4fa08-5563-4674-976c-de81a3a58148-kube-api-access-8dtcx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:50:57.261247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.261212    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:50:57.274223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:57.274194    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gwgjr"]
Nov 02 12:50:58.499001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:50:58.498969    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" path="/var/lib/kubelet/pods/7fe4fa08-5563-4674-976c-de81a3a58148/volumes"
Nov 02 12:51:09.744404 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:51:09.744364    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:51:12.064157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.064116    4928 generic.go:356] "Generic (PLEG): container finished" podID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerID="2c530cc34df9ee239389872c007f062c31d609b6a0cb7574808e1a1672cff89a" exitCode=0
Nov 02 12:51:12.064725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.064185    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerDied","Data":"2c530cc34df9ee239389872c007f062c31d609b6a0cb7574808e1a1672cff89a"}
Nov 02 12:51:12.064725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.064588    4928 scope.go:122] "RemoveContainer" containerID="2c530cc34df9ee239389872c007f062c31d609b6a0cb7574808e1a1672cff89a"
Nov 02 12:51:12.066349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.066309    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:12.066469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.066399    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:12.066469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:12.066449    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:13.074959 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:13.074926    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerStarted","Data":"24eb606aa35ccd9a7034c3609a95dfab8f5e191adeb0c9723fabbbf00883bb5b"}
Nov 02 12:51:13.593306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:13.593272    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:13.594753 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:13.594726    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 12:51:13.594898 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:13.594775    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 12:51:14.593081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:14.593043    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 12:51:14.593485 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:14.593106    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 12:51:14.860685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:14.860585    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:51:14.860929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:14.860886    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy" containerID="cri-o://d9f35cd6c6c1a77848d238435735b05ba6d489b4f08a07a8fe3f78029b80ae5b" gracePeriod=30
Nov 02 12:51:15.593176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:15.593139    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 12:51:15.593782 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:15.593751    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 12:51:16.593162 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:16.593127    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 12:51:16.593355 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:16.593180    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 12:51:16.924894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:16.924774    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:17.593164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:17.593127    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 12:51:17.593357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:17.593195    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 12:51:18.598552 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:18.598520    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:18.599709 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:18.599688    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:20.143283 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.143199    4928 generic.go:356] "Generic (PLEG): container finished" podID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerID="d9f35cd6c6c1a77848d238435735b05ba6d489b4f08a07a8fe3f78029b80ae5b" exitCode=0
Nov 02 12:51:20.143710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.143440    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" event={"ID":"609e74f0-7cf8-4324-8c10-ba5f4e16ba80","Type":"ContainerDied","Data":"d9f35cd6c6c1a77848d238435735b05ba6d489b4f08a07a8fe3f78029b80ae5b"}
Nov 02 12:51:20.143710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.143494    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn" event={"ID":"609e74f0-7cf8-4324-8c10-ba5f4e16ba80","Type":"ContainerDied","Data":"e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280"}
Nov 02 12:51:20.143710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.143520    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e125569ad3031ffde031c282b0fdb3e56258c8764b30880ddc60fcbc968280"
Nov 02 12:51:20.149342 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.149315    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.324542    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.324676    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.324735    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.324851    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.324978    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.325134    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.325194    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.325261    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.325325    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\") pod \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\" (UID: \"609e74f0-7cf8-4324-8c10-ba5f4e16ba80\") "
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.326630    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.326653    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.327792    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.328287    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:20.330039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.329099    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:20.330812 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.330366    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:20.330891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.330842    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:20.332422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.332399    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "kube-api-access-lrqzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:20.332548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.332532    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy" pod "609e74f0-7cf8-4324-8c10-ba5f4e16ba80" (UID: "609e74f0-7cf8-4324-8c10-ba5f4e16ba80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427617    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427649    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqzm\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-kube-api-access-lrqzm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427659    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427667    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427677    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427684    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427695    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427703    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:20.427711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:20.427711    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609e74f0-7cf8-4324-8c10-ba5f4e16ba80-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:21.152116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:21.152086    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"
Nov 02 12:51:21.222912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:21.222877    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:51:21.236458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:21.236425    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-l2rfn"]
Nov 02 12:51:22.499105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:22.499071    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" path="/var/lib/kubelet/pods/609e74f0-7cf8-4324-8c10-ba5f4e16ba80/volumes"
Nov 02 12:51:28.403672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.403584    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:51:28.404241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404222    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy"
Nov 02 12:51:28.404292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404242    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy"
Nov 02 12:51:28.404292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404265    4928 cpu_manager.go:414] "containerMap: removing container" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy"
Nov 02 12:51:28.404292 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404271    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy"
Nov 02 12:51:28.404387 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404380    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe4fa08-5563-4674-976c-de81a3a58148" containerName="istio-proxy"
Nov 02 12:51:28.404422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.404390    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="609e74f0-7cf8-4324-8c10-ba5f4e16ba80" containerName="istio-proxy"
Nov 02 12:51:28.409214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.409161    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.446295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.441197    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:51:28.453641 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.453589    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489064    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489139    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdmc\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489177    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489249    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489308    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489356    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489483    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489510    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.493584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.489609    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.495874 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.495771    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.509476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.509449    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591547    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591604    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591665    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591771    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591806    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdmc\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.591941    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592020    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592057    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598rc\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592098    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592165    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592213    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592250    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592279    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592417    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592446    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592553    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.592868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.592582    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.593863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.593794    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.595543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.594771    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.597862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.596054    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.597862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.596196    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.597862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.596215    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.598620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.598600    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.609725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.609678    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.619587 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.619549    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.623788 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.623754    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdmc\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.694338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694240    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598rc\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694302    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694367    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694390    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694552    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694638    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694666    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694706    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.694915 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.694754    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.695287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.695227    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.695693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.695668    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.696362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.696122    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.696523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.696159    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.696523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.696297    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.697598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.697575    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.697871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.697849    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.719284 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.719252    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.719439 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.719415    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-598rc\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:28.724340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.724312    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:28.816297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:28.815983    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:29.052564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.052536    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:51:29.054424 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:51:29.054395    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4f17b9_cab8_4c1e_a1a4_cb7d3dfb0e86.slice/crio-6062e0fdf2e78f69c4cce0041d436cf64e4809d8b163b0511a3202c79a678440 WatchSource:0}: Error finding container 6062e0fdf2e78f69c4cce0041d436cf64e4809d8b163b0511a3202c79a678440: Status 404 returned error can't find the container with id 6062e0fdf2e78f69c4cce0041d436cf64e4809d8b163b0511a3202c79a678440
Nov 02 12:51:29.057217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.057176    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.057297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.057243    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.057297 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.057273    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.148038 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:51:29.148004    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce59f0b_0892_4ccd_9c71_aa8bff9dbf21.slice/crio-68773d74e7d36f46ed20c2348db7aa8b52d16d5d655f2b331e304f4ce8bb2f28 WatchSource:0}: Error finding container 68773d74e7d36f46ed20c2348db7aa8b52d16d5d655f2b331e304f4ce8bb2f28: Status 404 returned error can't find the container with id 68773d74e7d36f46ed20c2348db7aa8b52d16d5d655f2b331e304f4ce8bb2f28
Nov 02 12:51:29.148935 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.148897    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:51:29.150891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.150853    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.151009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.150926    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.151009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.150955    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:29.227037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.226962    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" event={"ID":"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21","Type":"ContainerStarted","Data":"68773d74e7d36f46ed20c2348db7aa8b52d16d5d655f2b331e304f4ce8bb2f28"}
Nov 02 12:51:29.233841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.233782    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" event={"ID":"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86","Type":"ContainerStarted","Data":"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"}
Nov 02 12:51:29.234001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.233864    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" event={"ID":"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86","Type":"ContainerStarted","Data":"6062e0fdf2e78f69c4cce0041d436cf64e4809d8b163b0511a3202c79a678440"}
Nov 02 12:51:29.288320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.288253    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podStartSLOduration=1.288236003 podStartE2EDuration="1.288236003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:29.28647688 +0000 UTC m=+1486.935530839" watchObservedRunningTime="2026-11-02 12:51:29.288236003 +0000 UTC m=+1486.937289965"
Nov 02 12:51:29.724941 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:29.724904    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:30.278183 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.277327    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" event={"ID":"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21","Type":"ContainerStarted","Data":"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"}
Nov 02 12:51:30.401170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.401101    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podStartSLOduration=2.401077555 podStartE2EDuration="2.401077555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:30.363329981 +0000 UTC m=+1488.012383940" watchObservedRunningTime="2026-11-02 12:51:30.401077555 +0000 UTC m=+1488.050131520"
Nov 02 12:51:30.733901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.733659    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:51:30.733901 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.733737    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 12:51:30.816853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.816751    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:30.818302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.818151    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 12:51:30.818302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:30.818205    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 12:51:31.728853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:31.724895    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:51:31.728853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:31.724992    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 12:51:31.816762 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:31.816723    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 12:51:31.817195 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:31.816781    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 12:51:32.735282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:32.735240    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:51:32.735621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:32.735582    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 12:51:32.817515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:32.817477    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 12:51:32.818119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:32.818088    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 12:51:33.785198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:33.785093    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:51:33.785693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:33.785656    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 12:51:33.824310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:33.824276    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 12:51:33.824733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:33.824708    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 12:51:34.731852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:34.728955    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:51:34.731852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:34.729026    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 12:51:34.816762 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:34.816727    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 12:51:34.816981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:34.816913    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 12:51:36.184092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.184051    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:36.185240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.185181    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:36.186607 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.186583    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:51:36.229463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.229429    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:36.230053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.230033    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:36.232470 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:36.231729    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:51:39.755738 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:51:39.755697    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:51:42.815611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:42.815581    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:51:42.816222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:42.816122    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:51:43.315654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:43.315592    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:51:43.316249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:43.316211    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" containerID="cri-o://24eb606aa35ccd9a7034c3609a95dfab8f5e191adeb0c9723fabbbf00883bb5b" gracePeriod=30
Nov 02 12:51:43.505174 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:43.499799    4928 scope.go:122] "RemoveContainer" containerID="f3102325b67a98ac72ca05eeeb21e71060a9621e2d29667856debfababb3cb30"
Nov 02 12:51:43.552453 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:43.551308    4928 scope.go:122] "RemoveContainer" containerID="d9f35cd6c6c1a77848d238435735b05ba6d489b4f08a07a8fe3f78029b80ae5b"
Nov 02 12:51:44.207111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.207072    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:51:44.207634 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.207287    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" containerID="cri-o://66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc" gracePeriod=30
Nov 02 12:51:44.964341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.964307    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 12:51:44.970548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.970518    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:44.981030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.980999    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:51:44.985317 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.985291    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:51:44.985443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.985357    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:51:44.986369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.986347    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:44.999841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:44.999804    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 12:51:45.013270 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.013238    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:51:45.083225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083182    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083251    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083329    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.083404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083365    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083393    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083564 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083542    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb79\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.083597 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083579    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgtp\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083710 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.083787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083752    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083779    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.083951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083935    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.084004 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083974    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.084065 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.083999    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.084143 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.084124    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.084202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.084190    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.084253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.084241    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.084302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.084275    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.185786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.185749    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.185864    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb79\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.185899    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgtp\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186018 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.185974    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186028    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186053    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186192    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186243    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186285    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186301    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186374    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186440    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186476    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186477    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186481    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186509    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186525    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186570    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186592    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186645    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186688    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186721    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.186860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.186729    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.187243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.187181    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.187243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.187224    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.187364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.187279    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.187483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.187457    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.187546 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.187481    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.188941 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.188913    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.189080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.188973    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.189080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.189033    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.190411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.190389    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.211555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.211521    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.211972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.211808    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.211972 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.211927    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb79\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.212610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.212588    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgtp\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.288929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.286980    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:45.302380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.302301    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:45.647777 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:51:45.647738    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3808189d_ad8a_429c_8207_4b09f8ad9ce7.slice/crio-ad2f1a09f8289f1b5160888f8935bf7fc462f8f763b33e63468ae5a911b3c031 WatchSource:0}: Error finding container ad2f1a09f8289f1b5160888f8935bf7fc462f8f763b33e63468ae5a911b3c031: Status 404 returned error can't find the container with id ad2f1a09f8289f1b5160888f8935bf7fc462f8f763b33e63468ae5a911b3c031
Nov 02 12:51:45.648229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.648200    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 12:51:45.648403 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:51:45.648381    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae0248a_61fb_4d8b_9d0f_44a21278cf50.slice/crio-72764de59e94e96cf9cc0435f66a13cd4ea0c65b45010caf8e005c9c494847ea WatchSource:0}: Error finding container 72764de59e94e96cf9cc0435f66a13cd4ea0c65b45010caf8e005c9c494847ea: Status 404 returned error can't find the container with id 72764de59e94e96cf9cc0435f66a13cd4ea0c65b45010caf8e005c9c494847ea
Nov 02 12:51:45.655234 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.655190    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.655937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.655899    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.656036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.656021    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.657208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.655080    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.657329 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.657302    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.657471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.657401    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:51:45.659952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:45.659927    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:51:46.747848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.738774    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" event={"ID":"3808189d-ad8a-429c-8207-4b09f8ad9ce7","Type":"ContainerStarted","Data":"13ec57970e57aa2c422c70c897d432a4a3853792278ca6a80607b4861fe53752"}
Nov 02 12:51:46.747848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.738875    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" event={"ID":"3808189d-ad8a-429c-8207-4b09f8ad9ce7","Type":"ContainerStarted","Data":"ad2f1a09f8289f1b5160888f8935bf7fc462f8f763b33e63468ae5a911b3c031"}
Nov 02 12:51:46.773846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.766441    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" event={"ID":"dae0248a-61fb-4d8b-9d0f-44a21278cf50","Type":"ContainerStarted","Data":"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"}
Nov 02 12:51:46.773846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.766553    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" event={"ID":"dae0248a-61fb-4d8b-9d0f-44a21278cf50","Type":"ContainerStarted","Data":"72764de59e94e96cf9cc0435f66a13cd4ea0c65b45010caf8e005c9c494847ea"}
Nov 02 12:51:46.895846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.895241    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podStartSLOduration=2.895218499 podStartE2EDuration="2.895218499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:46.890268962 +0000 UTC m=+1504.539322946" watchObservedRunningTime="2026-11-02 12:51:46.895218499 +0000 UTC m=+1504.544272458"
Nov 02 12:51:46.947850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:46.946657    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podStartSLOduration=2.946637488 podStartE2EDuration="2.946637488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:46.94617383 +0000 UTC m=+1504.595227799" watchObservedRunningTime="2026-11-02 12:51:46.946637488 +0000 UTC m=+1504.595691447"
Nov 02 12:51:47.291983 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.291927    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:47.293441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.293407    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:47.293599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.293464    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:47.303001 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.302974    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:47.304339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.304316    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:47.304481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:47.304361    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:48.288364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.288322    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:48.288911 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.288386    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:48.303282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.303185    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:48.303282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.303239    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:48.806869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.806800    4928 generic.go:356] "Generic (PLEG): container finished" podID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerID="24eb606aa35ccd9a7034c3609a95dfab8f5e191adeb0c9723fabbbf00883bb5b" exitCode=0
Nov 02 12:51:48.807129 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.806871    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerDied","Data":"24eb606aa35ccd9a7034c3609a95dfab8f5e191adeb0c9723fabbbf00883bb5b"}
Nov 02 12:51:48.807218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.807172    4928 scope.go:122] "RemoveContainer" containerID="2c530cc34df9ee239389872c007f062c31d609b6a0cb7574808e1a1672cff89a"
Nov 02 12:51:48.890851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:48.890008    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:49.006340 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.006305    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.006509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.006381    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.006994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.006961    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007034    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007101 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007070    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007176 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007140    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007239    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007333    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007429    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\") pod \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\" (UID: \"4cbeeb8c-60ab-46ac-b868-b4c08f2be000\") "
Nov 02 12:51:49.007815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007331    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:49.008014 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007994    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.008014 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007579    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.008108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.007788    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.008257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.008236    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.008312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.008258    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.008312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.008272    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.008312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.008290    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.009426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.009394    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.010490 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.010466    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:49.012381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.012342    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:49.012992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.012952    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.013438 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.013403    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf" pod "4cbeeb8c-60ab-46ac-b868-b4c08f2be000" (UID: "4cbeeb8c-60ab-46ac-b868-b4c08f2be000"). InnerVolumeSpecName "kube-api-access-hrxvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:49.109709 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.109664    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxvf\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-kube-api-access-hrxvf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.109709 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.109707    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.109958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.109721    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.109958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.109734    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.109958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.109747    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbeeb8c-60ab-46ac-b868-b4c08f2be000-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.295918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.293354    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:49.295918 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.293419    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:49.304144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.303377    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:49.304144 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.303441    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:49.600727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.600637    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:51:49.600727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.600709    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:51:49.796399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.796307    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:51:49.816948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.816917    4928 generic.go:356] "Generic (PLEG): container finished" podID="318fa227-ec90-4c1c-8870-21803b84b28f" containerID="66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc" exitCode=0
Nov 02 12:51:49.817113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.816973    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"
Nov 02 12:51:49.817113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.816991    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerDied","Data":"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"}
Nov 02 12:51:49.817113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.817052    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km" event={"ID":"318fa227-ec90-4c1c-8870-21803b84b28f","Type":"ContainerDied","Data":"210b59618bc77806b0acce961471f7768521c7029af3966cb76683270ddac150"}
Nov 02 12:51:49.817113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.817081    4928 scope.go:122] "RemoveContainer" containerID="66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"
Nov 02 12:51:49.818875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.818853    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"
Nov 02 12:51:49.818978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.818853    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x" event={"ID":"4cbeeb8c-60ab-46ac-b868-b4c08f2be000","Type":"ContainerDied","Data":"83c6396b19e962a0e745a99c94ec5290abad7f61bf59b3d53d6ba996b7fb57ba"}
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863034    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863133    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863167    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863239    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863283    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863341    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863399    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863532    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.868027 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.863569    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\") pod \"318fa227-ec90-4c1c-8870-21803b84b28f\" (UID: \"318fa227-ec90-4c1c-8870-21803b84b28f\") "
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.874194    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.874507    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.874736    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.875683    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.875709    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.882885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.875724    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:49.893929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.890214    4928 scope.go:122] "RemoveContainer" containerID="28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"
Nov 02 12:51:49.893929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.892055    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:49.918190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.903589    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "kube-api-access-96csm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:49.918190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.906851    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:49.918190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.906919    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.918190 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.909020    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:49.977879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.932031    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token" pod "318fa227-ec90-4c1c-8870-21803b84b28f" (UID: "318fa227-ec90-4c1c-8870-21803b84b28f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.997416    4928 scope.go:122] "RemoveContainer" containerID="66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:51:49.998446    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc\": container with ID starting with 66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc not found: ID does not exist" containerID="66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.998485    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc"} err="failed to get container status \"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc\": rpc error: code = NotFound desc = could not find container \"66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc\": container with ID starting with 66f7a379daa757ea287049247aace8185866b94e85d9d830c35528bf93cf3ffc not found: ID does not exist"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:49.998512    4928 scope.go:122] "RemoveContainer" containerID="28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000743    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96csm\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-kube-api-access-96csm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000797    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000816    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318fa227-ec90-4c1c-8870-21803b84b28f-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000846    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318fa227-ec90-4c1c-8870-21803b84b28f-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000867    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318fa227-ec90-4c1c-8870-21803b84b28f-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.000884    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318fa227-ec90-4c1c-8870-21803b84b28f-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:51:50.000977    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b\": container with ID starting with 28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b not found: ID does not exist" containerID="28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.001009    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b"} err="failed to get container status \"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b\": rpc error: code = NotFound desc = could not find container \"28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b\": container with ID starting with 28452f75658e988ede8a296502ec918d9fe4ee35a4df45b2ea9a099806337e3b not found: ID does not exist"
Nov 02 12:51:50.007091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.001040    4928 scope.go:122] "RemoveContainer" containerID="24eb606aa35ccd9a7034c3609a95dfab8f5e191adeb0c9723fabbbf00883bb5b"
Nov 02 12:51:50.090975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.090878    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:51:50.096407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.096363    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3a281-openshift-default-6d6f445445-tgc9x"]
Nov 02 12:51:50.196092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.196056    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:51:50.199631 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.199594    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-tl2km"]
Nov 02 12:51:50.293751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.293213    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:50.293751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.293283    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:50.302605 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.302577    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:50.302999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.302626    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:50.503148 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.503112    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" path="/var/lib/kubelet/pods/318fa227-ec90-4c1c-8870-21803b84b28f/volumes"
Nov 02 12:51:50.503821 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:50.503802    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" path="/var/lib/kubelet/pods/4cbeeb8c-60ab-46ac-b868-b4c08f2be000/volumes"
Nov 02 12:51:51.294875 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:51.294811    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:51.295164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:51.295137    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:51.303327 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:51.303297    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:51.303707 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:51.303347    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:52.287938 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:52.287897    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 12:51:52.288132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:52.287967    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 12:51:52.303502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:52.303462    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 12:51:52.303905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:52.303535    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 12:51:53.878524 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:53.878487    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:53.879308 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:53.879282    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:53.880181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:53.880161    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 12:51:54.019360 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:54.019313    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:54.021897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:54.021874    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:54.024155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:54.024134    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:51:58.023851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.014480    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:51:58.121177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.121130    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy" containerID="cri-o://e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d" gracePeriod=30
Nov 02 12:51:58.164256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.164190    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:51:58.166781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.165323    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy" containerID="cri-o://699e19b38b0fa53fc9027304ef31d52584e6fabf215e2ebb56d15bc4a4c3d90b" gracePeriod=30
Nov 02 12:51:58.215976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.215452    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:51:58.216522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:51:58.216484    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy" containerID="cri-o://c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca" gracePeriod=30
Nov 02 12:52:03.293850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.293805    4928 generic.go:356] "Generic (PLEG): container finished" podID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerID="699e19b38b0fa53fc9027304ef31d52584e6fabf215e2ebb56d15bc4a4c3d90b" exitCode=0
Nov 02 12:52:03.294153 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.293876    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" event={"ID":"d8b768ba-8fae-4d31-af22-7bdd7aab289c","Type":"ContainerDied","Data":"699e19b38b0fa53fc9027304ef31d52584e6fabf215e2ebb56d15bc4a4c3d90b"}
Nov 02 12:52:03.629366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.629339    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:52:03.633058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.633035    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:52:03.721973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.721941    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722036    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722121    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722152    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722186    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722225    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722286    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722349    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722409    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722279    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.722530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722497    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722543    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722569    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722619    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.722728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722660    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.722728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722697    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.723005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722735    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.723005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722778    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.723005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722774    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\") pod \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\" (UID: \"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86\") "
Nov 02 12:52:03.723005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.722894    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\") pod \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\" (UID: \"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21\") "
Nov 02 12:52:03.723664 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.723639    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.723783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.723669    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.723783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.723757    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.724285 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.724261    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.725253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.725008    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.725892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.725592    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.726346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.726322    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.726716 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.726696    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.727215 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.727040    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.727749 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.727451    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.728392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.728326    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.728581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.728547    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.731021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.730131    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.731021 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.730710    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.731169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.731053    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.731473 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.731446    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc" pod "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" (UID: "fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86"). InnerVolumeSpecName "kube-api-access-2mdmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.731573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.731516    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.732815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.732792    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc" pod "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" (UID: "1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21"). InnerVolumeSpecName "kube-api-access-598rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.824891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824855    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.824891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824889    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824902    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824912    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824921    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-598rc\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-kube-api-access-598rc\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824930    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824938    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824946    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824953    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824964    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824972    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdmc\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-kube-api-access-2mdmc\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824982    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824990    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.824997    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.825005    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.825059 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.825013    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.905772 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:03.905743    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:52:04.028218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028175    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028281    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028404    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028483    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028538    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:04.028602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028568    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028571    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:04.028766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028626    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028657    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.028766 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.028713    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.029119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.029080    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:04.029309 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.029285    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:04.029667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.029644    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:04.029755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.029441    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\") pod \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\" (UID: \"d8b768ba-8fae-4d31-af22-7bdd7aab289c\") "
Nov 02 12:52:04.030403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.030358    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.030403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.030388    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.030562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.030405    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.030562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.030419    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.030562 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.030436    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.031550 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.031526    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:04.031732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.031702    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:04.031732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.031723    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:04.031880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.031749    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46" pod "d8b768ba-8fae-4d31-af22-7bdd7aab289c" (UID: "d8b768ba-8fae-4d31-af22-7bdd7aab289c"). InnerVolumeSpecName "kube-api-access-5zq46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:04.132239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.132157    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.132239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.132195    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zq46\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-kube-api-access-5zq46\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.132239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.132209    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.132239 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.132223    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b768ba-8fae-4d31-af22-7bdd7aab289c-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.302779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.302739    4928 generic.go:356] "Generic (PLEG): container finished" podID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerID="e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d" exitCode=0
Nov 02 12:52:04.303276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.302793    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"
Nov 02 12:52:04.303276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.302841    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" event={"ID":"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21","Type":"ContainerDied","Data":"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"}
Nov 02 12:52:04.303276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.302895    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx" event={"ID":"1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21","Type":"ContainerDied","Data":"68773d74e7d36f46ed20c2348db7aa8b52d16d5d655f2b331e304f4ce8bb2f28"}
Nov 02 12:52:04.303276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.302917    4928 scope.go:122] "RemoveContainer" containerID="e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"
Nov 02 12:52:04.304581 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.304558    4928 generic.go:356] "Generic (PLEG): container finished" podID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerID="c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca" exitCode=0
Nov 02 12:52:04.304695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.304612    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"
Nov 02 12:52:04.304768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.304700    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" event={"ID":"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86","Type":"ContainerDied","Data":"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"}
Nov 02 12:52:04.304768 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.304741    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv" event={"ID":"fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86","Type":"ContainerDied","Data":"6062e0fdf2e78f69c4cce0041d436cf64e4809d8b163b0511a3202c79a678440"}
Nov 02 12:52:04.307370 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.307343    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t" event={"ID":"d8b768ba-8fae-4d31-af22-7bdd7aab289c","Type":"ContainerDied","Data":"965825c6dfa3757483d6733d3eb70c169e2e69222f5c419edae2ed436d7fd9e3"}
Nov 02 12:52:04.307513 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.307349    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"
Nov 02 12:52:04.321652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.321627    4928 scope.go:122] "RemoveContainer" containerID="e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"
Nov 02 12:52:04.322096 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:04.322064    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d\": container with ID starting with e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d not found: ID does not exist" containerID="e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"
Nov 02 12:52:04.322206 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.322107    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d"} err="failed to get container status \"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d\": rpc error: code = NotFound desc = could not find container \"e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d\": container with ID starting with e95c45a977fa094c32728655017a2d5b81174d0ef726586b8e34f4e906a80a6d not found: ID does not exist"
Nov 02 12:52:04.322206 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.322130    4928 scope.go:122] "RemoveContainer" containerID="c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"
Nov 02 12:52:04.334540 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.334346    4928 scope.go:122] "RemoveContainer" containerID="c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"
Nov 02 12:52:04.334727 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:04.334702    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca\": container with ID starting with c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca not found: ID does not exist" containerID="c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"
Nov 02 12:52:04.334795 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.334733    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca"} err="failed to get container status \"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca\": rpc error: code = NotFound desc = could not find container \"c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca\": container with ID starting with c2d98b4c7a5b2f1ad9492ba677db5d7633646f939a7b5d798b19002339e923ca not found: ID does not exist"
Nov 02 12:52:04.334795 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.334750    4928 scope.go:122] "RemoveContainer" containerID="699e19b38b0fa53fc9027304ef31d52584e6fabf215e2ebb56d15bc4a4c3d90b"
Nov 02 12:52:04.396697 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.396612    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:52:04.403185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.403156    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-qk4rv"]
Nov 02 12:52:04.441867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.440039    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:52:04.444503 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.444471    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vn4qx"]
Nov 02 12:52:04.481929 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.481887    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:52:04.486549 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.486519    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vv85t"]
Nov 02 12:52:04.498320 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.498282    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" path="/var/lib/kubelet/pods/1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21/volumes"
Nov 02 12:52:04.498945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.498922    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" path="/var/lib/kubelet/pods/d8b768ba-8fae-4d31-af22-7bdd7aab289c/volumes"
Nov 02 12:52:04.499544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:04.499523    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" path="/var/lib/kubelet/pods/fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86/volumes"
Nov 02 12:52:07.353930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.352533    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:52:07.353930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.352758    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" podUID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" containerName="maas-api" containerID="cri-o://c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f" gracePeriod=30
Nov 02 12:52:07.695712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.695684    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:52:07.787879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.787818    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\") pod \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") "
Nov 02 12:52:07.788056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.787892    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\" (UniqueName: \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\") pod \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") "
Nov 02 12:52:07.788218 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.788195    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\") pod \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") "
Nov 02 12:52:07.788410 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.788394    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\") pod \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\" (UID: \"f70d5748-9c03-4014-994c-e8c3bc97ecd9\") "
Nov 02 12:52:07.789156 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.789121    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca" pod "f70d5748-9c03-4014-994c-e8c3bc97ecd9" (UID: "f70d5748-9c03-4014-994c-e8c3bc97ecd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:07.791294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.790900    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls" pod "f70d5748-9c03-4014-994c-e8c3bc97ecd9" (UID: "f70d5748-9c03-4014-994c-e8c3bc97ecd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:07.791294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.790971    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86" pod "f70d5748-9c03-4014-994c-e8c3bc97ecd9" (UID: "f70d5748-9c03-4014-994c-e8c3bc97ecd9"). InnerVolumeSpecName "kube-api-access-vhk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:07.791688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.791652    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs" pod "f70d5748-9c03-4014-994c-e8c3bc97ecd9" (UID: "f70d5748-9c03-4014-994c-e8c3bc97ecd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:07.896849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.894616    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.896849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.894656    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d5748-9c03-4014-994c-e8c3bc97ecd9-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.896849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.894673    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70d5748-9c03-4014-994c-e8c3bc97ecd9-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.896849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:07.894686    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhk86\" (UniqueName: \"kubernetes.io/projected/f70d5748-9c03-4014-994c-e8c3bc97ecd9-kube-api-access-vhk86\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:08.344061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.344027    4928 generic.go:356] "Generic (PLEG): container finished" podID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" containerID="c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f" exitCode=0
Nov 02 12:52:08.344242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.344091    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"
Nov 02 12:52:08.344242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.344111    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" event={"ID":"f70d5748-9c03-4014-994c-e8c3bc97ecd9","Type":"ContainerDied","Data":"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"}
Nov 02 12:52:08.344242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.344177    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz" event={"ID":"f70d5748-9c03-4014-994c-e8c3bc97ecd9","Type":"ContainerDied","Data":"e18bfd06f51d7d30fa019e253368c1683b3120547c78a29d75465d56b2f93531"}
Nov 02 12:52:08.344242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.344204    4928 scope.go:122] "RemoveContainer" containerID="c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"
Nov 02 12:52:08.361770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.361262    4928 scope.go:122] "RemoveContainer" containerID="c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"
Nov 02 12:52:08.361770 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:08.361601    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f\": container with ID starting with c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f not found: ID does not exist" containerID="c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"
Nov 02 12:52:08.361770 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.361673    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f"} err="failed to get container status \"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f\": rpc error: code = NotFound desc = could not find container \"c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f\": container with ID starting with c1400e072c75e1d2fff027305dde67fab982cc61960b8b51bfc943421b52a74f not found: ID does not exist"
Nov 02 12:52:08.402062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.402013    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:52:08.415310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.415271    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-729108-6d446b9855-f4mvz"]
Nov 02 12:52:08.498781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:08.498742    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" path="/var/lib/kubelet/pods/f70d5748-9c03-4014-994c-e8c3bc97ecd9/volumes"
Nov 02 12:52:09.731319 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:09.731273    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:52:13.025904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:13.025862    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:52:13.026367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:13.026122    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy" containerID="cri-o://002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26" gracePeriod=30
Nov 02 12:52:18.328726 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.328696    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:52:18.481853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.474649    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.481853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.474770    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.481853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.474813    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559295    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559350    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559389    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559434    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559572    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.559644 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559618    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\") pod \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\" (UID: \"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c\") "
Nov 02 12:52:18.560163 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.500391    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:18.560279 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.503035    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:18.560350 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.559122    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:18.560416 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.560134    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:18.560665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.560650    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:18.576539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.576495    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "kube-api-access-dwhdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:18.580855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.580514    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:18.608588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.585729    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609042    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609078    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609107    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609121    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609141    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhdf\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-kube-api-access-dwhdf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.609157    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.602458    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.602364    4928 generic.go:356] "Generic (PLEG): container finished" podID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerID="002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26" exitCode=0
Nov 02 12:52:18.610702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.601257    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:18.613905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.611340    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerDied","Data":"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"}
Nov 02 12:52:18.613905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.611418    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4" event={"ID":"8c57e715-0797-4a0d-ab7f-6e64b0e4d71c","Type":"ContainerDied","Data":"e2fd63ac5b04e8d81e51467f379bc051e88fedbfa27e012a447badad4750b868"}
Nov 02 12:52:18.613905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.611454    4928 scope.go:122] "RemoveContainer" containerID="002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"
Nov 02 12:52:18.613905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.611515    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy" pod "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" (UID: "8c57e715-0797-4a0d-ab7f-6e64b0e4d71c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:18.632099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.631966    4928 scope.go:122] "RemoveContainer" containerID="0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.658102    4928 scope.go:122] "RemoveContainer" containerID="002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:18.658542    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26\": container with ID starting with 002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26 not found: ID does not exist" containerID="002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.658579    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26"} err="failed to get container status \"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26\": rpc error: code = NotFound desc = could not find container \"002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26\": container with ID starting with 002985120d6e54a39ff2fcee25314ee4597c00a6a2fd34abf6870b716b103f26 not found: ID does not exist"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.658607    4928 scope.go:122] "RemoveContainer" containerID="0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:18.659372    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192\": container with ID starting with 0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192 not found: ID does not exist" containerID="0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"
Nov 02 12:52:18.659586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.659408    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192"} err="failed to get container status \"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192\": rpc error: code = NotFound desc = could not find container \"0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192\": container with ID starting with 0da2daea58ac5dcd50ff3facd2286e8103a9a70b24a5f95af724af25230aa192 not found: ID does not exist"
Nov 02 12:52:18.716181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.716099    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:18.716181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:18.716138    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:19.014893 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:19.013734    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:52:19.054166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:19.054118    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-729108-openshift-default-6ff5649589-xrnk4"]
Nov 02 12:52:20.186850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.184986    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.197942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.197891    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.314849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.305953    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.446908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.441868    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 12:52:20.523305 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.522394    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.544861 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.543992    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" path="/var/lib/kubelet/pods/8c57e715-0797-4a0d-ab7f-6e64b0e4d71c/volumes"
Nov 02 12:52:20.650898 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.650582    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 12:52:20.704906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.703941    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.925850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.919157    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.926074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:20.926034    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 12:52:21.186976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.186925    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:21.188330 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.188303    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:52:21.351376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.308903    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:21.351376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.309678    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:52:21.404398 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.404367    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 12:52:21.455423 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.455327    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 12:52:21.456315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.456290    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:52:21.620149 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.620116    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:52:21.708884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:21.708183    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 12:52:33.025011 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.024968    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:52:33.025599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025580    4928 cpu_manager.go:414] "containerMap: removing container" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.025599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025598    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025607    4928 cpu_manager.go:414] "containerMap: removing container" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025616    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025636    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" containerName="maas-api"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025641    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" containerName="maas-api"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025654    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025659    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025666    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025671    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025686    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025693    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025706    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025710    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025721    4928 cpu_manager.go:414] "containerMap: removing container" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025726    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025732    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.025731 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025736    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025753    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025758    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025911    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025927    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025941    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025951    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbeeb8c-60ab-46ac-b868-b4c08f2be000" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025963    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce59f0b-0892-4ccd-9c71-aa8bff9dbf21" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025973    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="318fa227-ec90-4c1c-8870-21803b84b28f" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025982    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4f17b9-cab8-4c1e-a1a4-cb7d3dfb0e86" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.025993    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70d5748-9c03-4014-994c-e8c3bc97ecd9" containerName="maas-api"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.026001    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c57e715-0797-4a0d-ab7f-6e64b0e4d71c" containerName="istio-proxy"
Nov 02 12:52:33.026506 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.026006    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b768ba-8fae-4d31-af22-7bdd7aab289c" containerName="istio-proxy"
Nov 02 12:52:33.029590 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.029572    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.059107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.059069    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:52:33.173602 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.173558    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ctp\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.173790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.173691    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.173920 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.173900    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.173992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.173975    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.174052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.174026    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.174115 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.174088    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.174229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.174205    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.174314 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.174298    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.174376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.174360    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.275942    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276036 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276004    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276113    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276155    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276232    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276256    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276324    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ctp\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276398    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276746    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276763    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276818    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.276892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.276863    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.277193 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.277172    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.278533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.278512    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.278779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.278759    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.300906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.300874    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.301058 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.300965    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ctp\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.349073 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.349035    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:33.680923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.680893    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:52:33.682289 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:52:33.682259    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaa51fb_4f4e_43d8_8f77_696f3067b111.slice/crio-5a1c6e8eb81ac2eb85bf31c172fd0dd5b40c30ae0704cd3a85e39473d6169252 WatchSource:0}: Error finding container 5a1c6e8eb81ac2eb85bf31c172fd0dd5b40c30ae0704cd3a85e39473d6169252: Status 404 returned error can't find the container with id 5a1c6e8eb81ac2eb85bf31c172fd0dd5b40c30ae0704cd3a85e39473d6169252
Nov 02 12:52:33.684769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.684726    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:52:33.684897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.684816    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:52:33.684973 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.684914    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:52:33.809886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.809850    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" event={"ID":"baaa51fb-4f4e-43d8-8f77-696f3067b111","Type":"ContainerStarted","Data":"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"}
Nov 02 12:52:33.810079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.809917    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" event={"ID":"baaa51fb-4f4e-43d8-8f77-696f3067b111","Type":"ContainerStarted","Data":"5a1c6e8eb81ac2eb85bf31c172fd0dd5b40c30ae0704cd3a85e39473d6169252"}
Nov 02 12:52:33.862868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:33.862778    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podStartSLOduration=1.862760809 podStartE2EDuration="1.862760809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:33.857457706 +0000 UTC m=+1551.506511665" watchObservedRunningTime="2026-11-02 12:52:33.862760809 +0000 UTC m=+1551.511814767"
Nov 02 12:52:34.360848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.350208    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:34.360848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.355952    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:34.360848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.356017    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:34.374138 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.370134    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:52:34.374138 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.370387    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" podUID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" containerName="maas-api" containerID="cri-o://f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb" gracePeriod=30
Nov 02 12:52:34.805617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.805574    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:52:34.877905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.877519    4928 generic.go:356] "Generic (PLEG): container finished" podID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" containerID="f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb" exitCode=0
Nov 02 12:52:34.889858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.878582    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"
Nov 02 12:52:34.889858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.878963    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" event={"ID":"5c2f20db-ed9a-4be6-a81b-56d409c95db1","Type":"ContainerDied","Data":"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"}
Nov 02 12:52:34.889858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.879014    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g" event={"ID":"5c2f20db-ed9a-4be6-a81b-56d409c95db1","Type":"ContainerDied","Data":"d5c8a0206087d3be719fe4036197d6412e1a0d4e00ccd9156e283398e5f6e1b6"}
Nov 02 12:52:34.889858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.879047    4928 scope.go:122] "RemoveContainer" containerID="f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"
Nov 02 12:52:34.923846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.905281    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") pod \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") "
Nov 02 12:52:34.923846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.905366    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\") pod \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") "
Nov 02 12:52:34.923846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.905409    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\" (UniqueName: \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\") pod \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") "
Nov 02 12:52:34.923846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.905583    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\") pod \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\" (UID: \"5c2f20db-ed9a-4be6-a81b-56d409c95db1\") "
Nov 02 12:52:34.923846 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.912386    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca" pod "5c2f20db-ed9a-4be6-a81b-56d409c95db1" (UID: "5c2f20db-ed9a-4be6-a81b-56d409c95db1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:34.949930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.942876    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m" pod "5c2f20db-ed9a-4be6-a81b-56d409c95db1" (UID: "5c2f20db-ed9a-4be6-a81b-56d409c95db1"). InnerVolumeSpecName "kube-api-access-vtd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:34.949930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.944960    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls" pod "5c2f20db-ed9a-4be6-a81b-56d409c95db1" (UID: "5c2f20db-ed9a-4be6-a81b-56d409c95db1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:34.949930 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.947967    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs" pod "5c2f20db-ed9a-4be6-a81b-56d409c95db1" (UID: "5c2f20db-ed9a-4be6-a81b-56d409c95db1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:34.989039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.981522    4928 scope.go:122] "RemoveContainer" containerID="f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"
Nov 02 12:52:34.989039 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:34.982095    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb\": container with ID starting with f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb not found: ID does not exist" containerID="f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"
Nov 02 12:52:34.989039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:34.982150    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb"} err="failed to get container status \"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb\": rpc error: code = NotFound desc = could not find container \"f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb\": container with ID starting with f4a99f483644f9cc8d3d3f3d3e90a1aec5cfd89370536ba553e8f4bde6655eeb not found: ID does not exist"
Nov 02 12:52:35.032853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.013735    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.032853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.013781    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2f20db-ed9a-4be6-a81b-56d409c95db1-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.032853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.013795    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f20db-ed9a-4be6-a81b-56d409c95db1-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.032853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.013813    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtd7m\" (UniqueName: \"kubernetes.io/projected/5c2f20db-ed9a-4be6-a81b-56d409c95db1-kube-api-access-vtd7m\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.261084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.261038    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:52:35.264431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.264401    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb24df-6f95cbd9c5-mhp9g"]
Nov 02 12:52:35.363851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.362951    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:35.363851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:35.363014    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:36.354847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.354340    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:36.354847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.354422    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:36.497721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.497687    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" path="/var/lib/kubelet/pods/5c2f20db-ed9a-4be6-a81b-56d409c95db1/volumes"
Nov 02 12:52:36.688461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.688374    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:52:36.688618 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.688576    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" podUID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" containerName="maas-api" containerID="cri-o://4f9967b0e5cda8baed5ae8e448356f9f08ce09cba0f9471c098af5242a3b2663" gracePeriod=30
Nov 02 12:52:36.918201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.916767    4928 generic.go:356] "Generic (PLEG): container finished" podID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" containerID="4f9967b0e5cda8baed5ae8e448356f9f08ce09cba0f9471c098af5242a3b2663" exitCode=0
Nov 02 12:52:36.918201 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:36.916963    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" event={"ID":"24ba0ed3-379c-49ec-843e-2c06a17f0ac3","Type":"ContainerDied","Data":"4f9967b0e5cda8baed5ae8e448356f9f08ce09cba0f9471c098af5242a3b2663"}
Nov 02 12:52:37.097034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.097003    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:52:37.192444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.192394    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\" (UniqueName: \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\") pod \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") "
Nov 02 12:52:37.192785 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.192758    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\") pod \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") "
Nov 02 12:52:37.193050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.193029    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\") pod \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") "
Nov 02 12:52:37.193268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.193250    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\") pod \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\" (UID: \"24ba0ed3-379c-49ec-843e-2c06a17f0ac3\") "
Nov 02 12:52:37.206872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.200713    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca" pod "24ba0ed3-379c-49ec-843e-2c06a17f0ac3" (UID: "24ba0ed3-379c-49ec-843e-2c06a17f0ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:37.208425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.208399    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.272358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.272268    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls" pod "24ba0ed3-379c-49ec-843e-2c06a17f0ac3" (UID: "24ba0ed3-379c-49ec-843e-2c06a17f0ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:37.274383 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.274357    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4" pod "24ba0ed3-379c-49ec-843e-2c06a17f0ac3" (UID: "24ba0ed3-379c-49ec-843e-2c06a17f0ac3"). InnerVolumeSpecName "kube-api-access-4mdr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:37.276738 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.276699    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs" pod "24ba0ed3-379c-49ec-843e-2c06a17f0ac3" (UID: "24ba0ed3-379c-49ec-843e-2c06a17f0ac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:37.312850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.310682    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdr4\" (UniqueName: \"kubernetes.io/projected/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-kube-api-access-4mdr4\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.312850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.310740    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.312850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.310759    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ba0ed3-379c-49ec-843e-2c06a17f0ac3-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.364937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.364448    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:37.364937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.364518    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:37.952853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.942299    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b" event={"ID":"24ba0ed3-379c-49ec-843e-2c06a17f0ac3","Type":"ContainerDied","Data":"5d7ca2443f8e1c7bcbec9264b40e93b6e2c59daf46f89030813f3c8a49df7739"}
Nov 02 12:52:37.952853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.942395    4928 scope.go:122] "RemoveContainer" containerID="4f9967b0e5cda8baed5ae8e448356f9f08ce09cba0f9471c098af5242a3b2663"
Nov 02 12:52:37.952853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:37.949938    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"
Nov 02 12:52:38.014771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:38.014696    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:52:38.059599 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:38.059504    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-299e74-5666d7b7f9-5l62b"]
Nov 02 12:52:38.352908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:38.352246    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:38.352908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:38.352309    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:38.498649 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:38.498614    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" path="/var/lib/kubelet/pods/24ba0ed3-379c-49ec-843e-2c06a17f0ac3/volumes"
Nov 02 12:52:39.357347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:39.357289    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:39.357923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:39.357353    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:39.923404 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:39.923367    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:52:40.357867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:40.357373    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:40.357867 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:40.357453    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:41.377848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.368397    4928 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:41.377848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.368465    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 12:52:41.406851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.405947    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.627909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.626969    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.656243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.656186    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.704109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.703985    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.711960 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.711905    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.912732 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:41.908553    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:42.488976 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:42.488932    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:42.499677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:42.499645    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:42.499858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:42.499755    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:52:42.743161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:42.743078    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:52:42.743389 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:42.743358    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy" containerID="cri-o://37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0" gracePeriod=30
Nov 02 12:52:48.063728 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.063698    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:52:48.123078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123039    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123127    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123334 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123294    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123406 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123395    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123499    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123561    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:48.123669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123575    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:48.123793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123691    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123711    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:48.123793 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123761    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123810    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123871    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\") pod \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\" (UID: \"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27\") "
Nov 02 12:52:48.123961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.123877    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:48.124088 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124028    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:48.124512 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124489    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.124610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124515    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.124610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124529    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.124610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124542    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.124610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.124555    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.125708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.125668    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:48.125869 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.125800    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:48.126091 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.126066    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:48.126208 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.126155    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8" pod "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" (UID: "e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27"). InnerVolumeSpecName "kube-api-access-fm5d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:48.161247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.161206    4928 generic.go:356] "Generic (PLEG): container finished" podID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerID="37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0" exitCode=0
Nov 02 12:52:48.161405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.161283    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerDied","Data":"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"}
Nov 02 12:52:48.161405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.161339    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl" event={"ID":"e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27","Type":"ContainerDied","Data":"9aeaaacb45416edcbd33c69601061783cf56385f6e6193fb7d580ffad92eb962"}
Nov 02 12:52:48.161405 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.161371    4928 scope.go:122] "RemoveContainer" containerID="37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"
Nov 02 12:52:48.161606 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.161550    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"
Nov 02 12:52:48.182787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.182645    4928 scope.go:122] "RemoveContainer" containerID="883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"
Nov 02 12:52:48.197310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.197282    4928 scope.go:122] "RemoveContainer" containerID="37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"
Nov 02 12:52:48.197810 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:48.197654    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0\": container with ID starting with 37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0 not found: ID does not exist" containerID="37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"
Nov 02 12:52:48.197810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.197697    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0"} err="failed to get container status \"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0\": rpc error: code = NotFound desc = could not find container \"37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0\": container with ID starting with 37f2c0ecc81b04f2d43836e0f5336fcd086d692cdeec8c0aba7affb4fbc031f0 not found: ID does not exist"
Nov 02 12:52:48.197810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.197723    4928 scope.go:122] "RemoveContainer" containerID="883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"
Nov 02 12:52:48.198110 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:52:48.198043    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31\": container with ID starting with 883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31 not found: ID does not exist" containerID="883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"
Nov 02 12:52:48.198110 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.198065    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31"} err="failed to get container status \"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31\": rpc error: code = NotFound desc = could not find container \"883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31\": container with ID starting with 883ea82f3da3fcfab38ef7bdaf4aa65f0f22892d8c1030001b77ae94d9c4fe31 not found: ID does not exist"
Nov 02 12:52:48.226089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.226064    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.226185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.226092    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5d8\" (UniqueName: \"kubernetes.io/projected/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-kube-api-access-fm5d8\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.226185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.226108    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.226185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.226122    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:48.237792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.237764    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:52:48.240956 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.240934    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-299e74-openshift-default-5dfb594b9c-zjdwl"]
Nov 02 12:52:48.499866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:48.499818    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" path="/var/lib/kubelet/pods/e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27/volumes"
Nov 02 12:52:51.054548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.054504    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"]
Nov 02 12:52:51.055371 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055348    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:52:51.055477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055372    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:52:51.055477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055423    4928 cpu_manager.go:414] "containerMap: removing container" podUID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" containerName="maas-api"
Nov 02 12:52:51.055477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055432    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" containerName="maas-api"
Nov 02 12:52:51.055477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055452    4928 cpu_manager.go:414] "containerMap: removing container" podUID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" containerName="maas-api"
Nov 02 12:52:51.055477 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055461    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" containerName="maas-api"
Nov 02 12:52:51.055737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055638    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2f20db-ed9a-4be6-a81b-56d409c95db1" containerName="maas-api"
Nov 02 12:52:51.055737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055651    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:52:51.055737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055663    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ba0ed3-379c-49ec-843e-2c06a17f0ac3" containerName="maas-api"
Nov 02 12:52:51.055737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.055679    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:52:51.060895 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.060870    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.081013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.080984    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"]
Nov 02 12:52:51.089574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.089542    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:51.089803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.089780    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:51.163652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.163606    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwc7\" (UniqueName: \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.163925 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.163898    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.164106 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.164076    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.265910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.265874    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.266107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.265977    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.266107 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.266014    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwc7\" (UniqueName: \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.266683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.266657    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.268318 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.268282    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.291418 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.291390    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwc7\" (UniqueName: \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\") pod \"maas-api-revoke-sub-ad426528-4zrw5\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.375877 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.375766    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:51.712813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:51.712777    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"]
Nov 02 12:52:51.759868 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:52:51.757955    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda124b66_675c_4740_9f8a_bfa5b02e3d40.slice/crio-248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824 WatchSource:0}: Error finding container 248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824: Status 404 returned error can't find the container with id 248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824
Nov 02 12:52:52.197077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:52.197043    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5" event={"ID":"da124b66-675c-4740-9f8a-bfa5b02e3d40","Type":"ContainerStarted","Data":"248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824"}
Nov 02 12:52:53.206181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:53.206087    4928 generic.go:356] "Generic (PLEG): container finished" podID="da124b66-675c-4740-9f8a-bfa5b02e3d40" containerID="c250e366d64355015830aa21e3ec74473e5e0b965b93bd3975d9dbf040e8beaa" exitCode=0
Nov 02 12:52:53.206567 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:53.206176    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5" event={"ID":"da124b66-675c-4740-9f8a-bfa5b02e3d40","Type":"ContainerDied","Data":"c250e366d64355015830aa21e3ec74473e5e0b965b93bd3975d9dbf040e8beaa"}
Nov 02 12:52:54.431717 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.431338    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:54.513462 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.513409    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\") pod \"da124b66-675c-4740-9f8a-bfa5b02e3d40\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") "
Nov 02 12:52:54.513949 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.513920    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\") pod \"da124b66-675c-4740-9f8a-bfa5b02e3d40\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") "
Nov 02 12:52:54.514600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.514577    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\" (UniqueName: \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\") pod \"da124b66-675c-4740-9f8a-bfa5b02e3d40\" (UID: \"da124b66-675c-4740-9f8a-bfa5b02e3d40\") "
Nov 02 12:52:54.515561 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.514171    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca" pod "da124b66-675c-4740-9f8a-bfa5b02e3d40" (UID: "da124b66-675c-4740-9f8a-bfa5b02e3d40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:54.522020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.521945    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca" pod "da124b66-675c-4740-9f8a-bfa5b02e3d40" (UID: "da124b66-675c-4740-9f8a-bfa5b02e3d40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:54.522020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.521963    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7" pod "da124b66-675c-4740-9f8a-bfa5b02e3d40" (UID: "da124b66-675c-4740-9f8a-bfa5b02e3d40"). InnerVolumeSpecName "kube-api-access-xwwc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:54.621315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.617159    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwc7\" (UniqueName: \"kubernetes.io/projected/da124b66-675c-4740-9f8a-bfa5b02e3d40-kube-api-access-xwwc7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:54.621315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.617204    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:54.621315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:54.617223    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da124b66-675c-4740-9f8a-bfa5b02e3d40-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:52:55.246855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:55.246165    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5" event={"ID":"da124b66-675c-4740-9f8a-bfa5b02e3d40","Type":"ContainerDied","Data":"248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824"}
Nov 02 12:52:55.246855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:55.246229    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="248d05d641cfc3315c76bb893e0a7938bc4817ed065e61f909b5fe1e60e07824"
Nov 02 12:52:55.246855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:55.246299    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"
Nov 02 12:52:55.902854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:55.888927    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:55.924856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:55.915066    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.040851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.039974    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.548851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.516816    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.644849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.640939    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.644849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.642537    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.665849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.658920    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.704950 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.704903    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.717666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.717627    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.939993 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.939890    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.987850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.939890    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.000243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:56.963148    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:52:57.052002 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.051965    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:52:57.188181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.052658    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.666851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.662561    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:52:57.666851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.662665    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:52:57.682961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.672658    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.695770    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.722620    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.723166    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.723255    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.725909    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.748854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.726600    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:52:57.843850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.842937    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.882848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.878286    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.882848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.878681    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:52:57.960294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:57.960197    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 12:52:58.002277 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.002232    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.043695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.043150    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:52:58.051081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.051054    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 12:52:58.104232 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.094995    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:52:58.184158 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.184129    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:52:58.206534 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:52:58.206509    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:53:09.732910 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:53:09.732872    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:53:10.584185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584149    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"]
Nov 02 12:53:10.584745 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584731    4928 cpu_manager.go:414] "containerMap: removing container" podUID="da124b66-675c-4740-9f8a-bfa5b02e3d40" containerName="revoke-keys"
Nov 02 12:53:10.584790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584746    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da124b66-675c-4740-9f8a-bfa5b02e3d40" containerName="revoke-keys"
Nov 02 12:53:10.584790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584758    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:53:10.584790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584763    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f069f0-27c5-4a7d-b75f-0cd9f82d4a27" containerName="istio-proxy"
Nov 02 12:53:10.584902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.584881    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="da124b66-675c-4740-9f8a-bfa5b02e3d40" containerName="revoke-keys"
Nov 02 12:53:10.588570 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.588551    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.612796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.612749    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"]
Nov 02 12:53:10.646885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.646777    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.647078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.646889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.647150 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.647080    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsjj\" (UniqueName: \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.690394 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.690356    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:53:10.690666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.690643    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:53:10.748676 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.748636    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsjj\" (UniqueName: \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.749186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.748808    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.749186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.748860    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.749468 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.749445    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.751444 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.751413    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.774488 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.774451    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsjj\" (UniqueName: \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\") pod \"maas-api-revoke-sub-3b263cf5-dn7hs\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:10.903097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:10.902995    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:11.269797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:11.269753    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"]
Nov 02 12:53:11.270964 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:11.270924    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff6110e_a0c8_456e_9ac3_40a200310572.slice/crio-de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371 WatchSource:0}: Error finding container de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371: Status 404 returned error can't find the container with id de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371
Nov 02 12:53:12.141141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:12.141095    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs" event={"ID":"2ff6110e-a0c8-456e-9ac3-40a200310572","Type":"ContainerStarted","Data":"49fabcb042624d7dc1718ecaf8f5e98ee4271e20f432a9e5dbeea2ca760a76f5"}
Nov 02 12:53:12.141560 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:12.141164    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs" event={"ID":"2ff6110e-a0c8-456e-9ac3-40a200310572","Type":"ContainerStarted","Data":"de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371"}
Nov 02 12:53:12.189358 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:12.189296    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs" podStartSLOduration=2.189278552 podStartE2EDuration="2.189278552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:12.185269387 +0000 UTC m=+1589.834323345" watchObservedRunningTime="2026-11-02 12:53:12.189278552 +0000 UTC m=+1589.838332512"
Nov 02 12:53:13.250866 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:13.250804    4928 generic.go:356] "Generic (PLEG): container finished" podID="2ff6110e-a0c8-456e-9ac3-40a200310572" containerID="49fabcb042624d7dc1718ecaf8f5e98ee4271e20f432a9e5dbeea2ca760a76f5" exitCode=0
Nov 02 12:53:13.251387 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:13.250978    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs" event={"ID":"2ff6110e-a0c8-456e-9ac3-40a200310572","Type":"ContainerDied","Data":"49fabcb042624d7dc1718ecaf8f5e98ee4271e20f432a9e5dbeea2ca760a76f5"}
Nov 02 12:53:14.429921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.429891    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:14.522910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.522768    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\") pod \"2ff6110e-a0c8-456e-9ac3-40a200310572\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") "
Nov 02 12:53:14.522910 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.522865    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\" (UniqueName: \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\") pod \"2ff6110e-a0c8-456e-9ac3-40a200310572\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") "
Nov 02 12:53:14.524180 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.522961    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\") pod \"2ff6110e-a0c8-456e-9ac3-40a200310572\" (UID: \"2ff6110e-a0c8-456e-9ac3-40a200310572\") "
Nov 02 12:53:14.524180 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.524003    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca" pod "2ff6110e-a0c8-456e-9ac3-40a200310572" (UID: "2ff6110e-a0c8-456e-9ac3-40a200310572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:14.527085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.526965    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj" pod "2ff6110e-a0c8-456e-9ac3-40a200310572" (UID: "2ff6110e-a0c8-456e-9ac3-40a200310572"). InnerVolumeSpecName "kube-api-access-mwsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:14.527385 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.527355    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca" pod "2ff6110e-a0c8-456e-9ac3-40a200310572" (UID: "2ff6110e-a0c8-456e-9ac3-40a200310572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:14.624811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.624771    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:53:14.624811 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.624813    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsjj\" (UniqueName: \"kubernetes.io/projected/2ff6110e-a0c8-456e-9ac3-40a200310572-kube-api-access-mwsjj\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:53:14.625072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:14.624869    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6110e-a0c8-456e-9ac3-40a200310572-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:53:15.271680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:15.271640    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"
Nov 02 12:53:15.271680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:15.271663    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs" event={"ID":"2ff6110e-a0c8-456e-9ac3-40a200310572","Type":"ContainerDied","Data":"de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371"}
Nov 02 12:53:15.271932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:15.271723    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de38757d98aa8c5767ecb5fdd0df90b013b2295778ed56dcfb7e31ecf6b82371"
Nov 02 12:53:17.057978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.057924    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.206951 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.206897    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.221323 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.221280    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.221914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.221878    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:53:17.329904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.322774    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.973978 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:17.973923    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.017961 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.017911    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.073009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.072962    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.075037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.075012    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:53:18.135847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.119121    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.164166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.164117    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.227495 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.227380    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.245853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.242647    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.412466 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.262535    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:53:18.412654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.412530    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:53:18.412816 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.293896    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.413189 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.413165    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:53:18.421850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:18.415127    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.017050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.017023    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 12:53:19.112974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.095923    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.130017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.129973    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.131366 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.131205    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:53:19.133122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.133104    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 12:53:19.370604 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.358393    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 12:53:19.426642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.425455    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 12:53:19.430672 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:19.430258    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 12:53:22.125796 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.125761    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:53:22.126374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.126358    4928 cpu_manager.go:414] "containerMap: removing container" podUID="2ff6110e-a0c8-456e-9ac3-40a200310572" containerName="revoke-keys"
Nov 02 12:53:22.126374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.126374    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff6110e-a0c8-456e-9ac3-40a200310572" containerName="revoke-keys"
Nov 02 12:53:22.126501 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.126492    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff6110e-a0c8-456e-9ac3-40a200310572" containerName="revoke-keys"
Nov 02 12:53:22.130537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.130507    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.215905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.215869    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:53:22.233450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233410    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233481    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233569    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233612    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233671 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233640    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233726    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233759    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s69d\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233807    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.233933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.233878    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.335556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335516    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.335754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335577    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s69d\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.335754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335640    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.335754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335702    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.335979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335926    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.335989    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336003    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336020    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336084    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336114    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336136    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336318    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336393    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.336687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.336658    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.338250 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.338229    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.338591 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.338572    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.360053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.360022    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.360247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.360226    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s69d\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.448573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.448485    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:22.781763 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.781723    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:53:22.782465 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:22.782436    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61699b6_c8e7_41a4_bbd9_7cbbf8a490ad.slice/crio-8faa09686237445f14257396201245b24e0b8299b271e5ecaac3746bcbf7a321 WatchSource:0}: Error finding container 8faa09686237445f14257396201245b24e0b8299b271e5ecaac3746bcbf7a321: Status 404 returned error can't find the container with id 8faa09686237445f14257396201245b24e0b8299b271e5ecaac3746bcbf7a321
Nov 02 12:53:22.784990 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.784951    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:22.785084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.785018    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:22.785084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:22.785050    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:23.759665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:23.759620    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" event={"ID":"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad","Type":"ContainerStarted","Data":"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"}
Nov 02 12:53:23.760207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:23.759701    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" event={"ID":"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad","Type":"ContainerStarted","Data":"8faa09686237445f14257396201245b24e0b8299b271e5ecaac3746bcbf7a321"}
Nov 02 12:53:23.853849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:23.853130    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podStartSLOduration=1.853110457 podStartE2EDuration="1.853110457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:23.83473484 +0000 UTC m=+1601.483788799" watchObservedRunningTime="2026-11-02 12:53:23.853110457 +0000 UTC m=+1601.502164417"
Nov 02 12:53:24.452922 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:24.452887    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:24.454862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:24.453640    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:24.454862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:24.454575    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:25.450209 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:25.450163    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:25.450843 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:25.450790    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:26.449440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:26.449402    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:26.449739 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:26.449695    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:27.451849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:27.451149    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:27.451849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:27.451213    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:28.450220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:28.450179    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:28.450528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:28.450499    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:29.452628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.452590    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 12:53:29.453261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.453226    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 12:53:29.728306 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.728225    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 12:53:29.732695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.732676    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.766458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.766421    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 12:53:29.798122 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.798080    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a6f2c0\""
Nov 02 12:53:29.841008 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.840958    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.841221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.841105    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6pp\" (UniqueName: \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.942936 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.942887    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.944344 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.944118    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.944560 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.944539    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6pp\" (UniqueName: \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:29.980553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:29.980475    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6pp\" (UniqueName: \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\") pod \"payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:30.049113 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.049068    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:30.421999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.421961    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 12:53:30.486118 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.486091    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:30.487932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.487912    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:30.488942 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.488901    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:53:30.846847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.845964    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" event={"ID":"a74e4841-d2ba-4a0d-82ab-a047e71ac45d","Type":"ContainerStarted","Data":"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"}
Nov 02 12:53:30.846847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.846036    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" event={"ID":"a74e4841-d2ba-4a0d-82ab-a047e71ac45d","Type":"ContainerStarted","Data":"7062b4c37761ea67def5bc6dae2ced4e543289d4fb09d4734416f8d0c9b77a5c"}
Nov 02 12:53:30.846847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.846252    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:30.903716 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:30.903659    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" podStartSLOduration=1.903645059 podStartE2EDuration="1.903645059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:30.901540646 +0000 UTC m=+1608.550594595" watchObservedRunningTime="2026-11-02 12:53:30.903645059 +0000 UTC m=+1608.552699017"
Nov 02 12:53:32.649585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.649523    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 12:53:32.676326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.675307    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.705853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.705492    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ed56b9\""
Nov 02 12:53:32.705853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.705792    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 12:53:32.754836 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.754782    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ed56b9-dockercfg-vj2x9\""
Nov 02 12:53:32.808800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.808765    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.809075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.809056    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djv8q\" (UniqueName: \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.910440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.910353    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djv8q\" (UniqueName: \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.910440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.910436    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.911263 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.911242    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:32.937683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.937645    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djv8q\" (UniqueName: \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\") pod \"payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:33.000013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:32.999974    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:33.324554 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.324522    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 12:53:33.325935 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:33.325893    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da0be6d_3387_4118_8599_25e9a8026a4a.slice/crio-fddbc5d58ac645d5a97a8508f12de468a243cb9c51be784d1b8d95de20c33582 WatchSource:0}: Error finding container fddbc5d58ac645d5a97a8508f12de468a243cb9c51be784d1b8d95de20c33582: Status 404 returned error can't find the container with id fddbc5d58ac645d5a97a8508f12de468a243cb9c51be784d1b8d95de20c33582
Nov 02 12:53:33.880037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.879997    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" event={"ID":"2da0be6d-3387-4118-8599-25e9a8026a4a","Type":"ContainerStarted","Data":"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"}
Nov 02 12:53:33.880481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.880054    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" event={"ID":"2da0be6d-3387-4118-8599-25e9a8026a4a","Type":"ContainerStarted","Data":"fddbc5d58ac645d5a97a8508f12de468a243cb9c51be784d1b8d95de20c33582"}
Nov 02 12:53:33.880481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.880217    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:33.926863 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.926781    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" podStartSLOduration=1.9267604459999998 podStartE2EDuration="1.926760446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:33.925635159 +0000 UTC m=+1611.574689120" watchObservedRunningTime="2026-11-02 12:53:33.926760446 +0000 UTC m=+1611.575814405"
Nov 02 12:53:33.968043 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.968007    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 12:53:33.972575 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.972548    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:33.999880 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:33.999851    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e7561a\""
Nov 02 12:53:34.000169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.000150    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e7561a-dockercfg-rppv6\""
Nov 02 12:53:34.000908 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.000886    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 12:53:34.115853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.083215    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 12:53:34.125280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.125245    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.128003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.127971    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.128199 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.128181    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pznpz\" (UniqueName: \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.148433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.143646    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 12:53:34.248547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.248257    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmwp\" (UniqueName: \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.248547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.248388    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.248547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.248462    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.248547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.248513    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pznpz\" (UniqueName: \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.251090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.249971    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.276722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.276681    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pznpz\" (UniqueName: \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\") pod \"payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.288476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.288449    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.351786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.350229    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.351786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.350494    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmwp\" (UniqueName: \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.351786 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.351729    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.377426 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.377389    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmwp\" (UniqueName: \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\") pod \"payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.565209 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.565173    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:34.605539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.605499    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 12:53:34.605926 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:34.605899    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a385f5_ac60_4ee4_9c30_4356b4a9f913.slice/crio-57a69cdcbbde2e8dcf1c21fe1357cf6a0a302bb8d77fc1d43694dce2faf10b36 WatchSource:0}: Error finding container 57a69cdcbbde2e8dcf1c21fe1357cf6a0a302bb8d77fc1d43694dce2faf10b36: Status 404 returned error can't find the container with id 57a69cdcbbde2e8dcf1c21fe1357cf6a0a302bb8d77fc1d43694dce2faf10b36
Nov 02 12:53:34.883790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.883757    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 12:53:34.884170 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:34.883872    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6504ebd1_aaa7_4bdc_8105_6b8991468c64.slice/crio-82ee612f88333471a729c8628aea2ea852532df8f4daee1f86eb37785affe609 WatchSource:0}: Error finding container 82ee612f88333471a729c8628aea2ea852532df8f4daee1f86eb37785affe609: Status 404 returned error can't find the container with id 82ee612f88333471a729c8628aea2ea852532df8f4daee1f86eb37785affe609
Nov 02 12:53:34.899888 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.899812    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" event={"ID":"11a385f5-ac60-4ee4-9c30-4356b4a9f913","Type":"ContainerStarted","Data":"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"}
Nov 02 12:53:34.899999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.899955    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:34.899999 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.899995    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" event={"ID":"11a385f5-ac60-4ee4-9c30-4356b4a9f913","Type":"ContainerStarted","Data":"57a69cdcbbde2e8dcf1c21fe1357cf6a0a302bb8d77fc1d43694dce2faf10b36"}
Nov 02 12:53:34.957047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:34.956971    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" podStartSLOduration=1.956933681 podStartE2EDuration="1.956933681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:34.951072944 +0000 UTC m=+1612.600126904" watchObservedRunningTime="2026-11-02 12:53:34.956933681 +0000 UTC m=+1612.605987639"
Nov 02 12:53:35.909123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:35.909008    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" event={"ID":"6504ebd1-aaa7-4bdc-8105-6b8991468c64","Type":"ContainerStarted","Data":"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"}
Nov 02 12:53:35.909577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:35.909156    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" event={"ID":"6504ebd1-aaa7-4bdc-8105-6b8991468c64","Type":"ContainerStarted","Data":"82ee612f88333471a729c8628aea2ea852532df8f4daee1f86eb37785affe609"}
Nov 02 12:53:35.909577 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:35.909331    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:35.954702 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:35.954645    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" podStartSLOduration=1.954632704 podStartE2EDuration="1.954632704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:35.951735506 +0000 UTC m=+1613.600789465" watchObservedRunningTime="2026-11-02 12:53:35.954632704 +0000 UTC m=+1613.603686659"
Nov 02 12:53:39.731693 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:53:39.731654    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:53:41.857151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:41.857114    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 12:53:42.875996 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.875956    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 12:53:42.881396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.881363    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:42.912186 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.912131    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a6f2c0-serving-cert\""
Nov 02 12:53:42.913393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.913359    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8kr7\" (UniqueName: \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:42.915272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.915246    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:42.915553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.915531    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:42.918803 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.918778    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:42.938070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:42.936091    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 12:53:43.020637 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.020601    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8kr7\" (UniqueName: \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.020850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.020672    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.020850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.020714    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.020850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.020781    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.021574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.021542    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.023376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.023351    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.023467 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.023405    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.106010 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.105972    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8kr7\" (UniqueName: \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\") pod \"maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.196279 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.196192    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:43.521207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.521172    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 12:53:43.522208 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:43.522173    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ed7422_66b4_4198_9b85_956064954579.slice/crio-447b84793320b5ba29e29bd93e67ccd5d773aa2e9518b18a294c522d4ce15139 WatchSource:0}: Error finding container 447b84793320b5ba29e29bd93e67ccd5d773aa2e9518b18a294c522d4ce15139: Status 404 returned error can't find the container with id 447b84793320b5ba29e29bd93e67ccd5d773aa2e9518b18a294c522d4ce15139
Nov 02 12:53:43.524682 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.524658    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:53:43.993921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:43.993840    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" event={"ID":"91ed7422-66b4-4198-9b85-956064954579","Type":"ContainerStarted","Data":"447b84793320b5ba29e29bd93e67ccd5d773aa2e9518b18a294c522d4ce15139"}
Nov 02 12:53:44.738424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.738386    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:53:44.743970 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.743942    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.754404 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.754375    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:53:44.758417 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.758393    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.787906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.787872    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:53:44.795723 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.795679    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:53:44.858197 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858162    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858197 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858207    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.858425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858291    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.858425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858327    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858395    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.858617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858433    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858527    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.858617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858590    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858627    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858702    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858726    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858745    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9gxx\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.858784 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858761    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m25t\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.859039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858898    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.859039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858934    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.859039 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.858954    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.859154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.859038    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.859154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.859134    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.903092 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.903068    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 12:53:44.960213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960170    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960222    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960252    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960349    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960404    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960466    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960496    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960522    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960579    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960657 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960646    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960693    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960747    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960754    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960773    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960850    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960861    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960926    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.960979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960960    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9gxx\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.961407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.960988    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m25t\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.961407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.961084    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.961407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.961128    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.961407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.961227    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.961624 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.961488    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.963638 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.962559    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.963779 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.963747    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.965236 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.964999    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.965395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.965372    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.965583 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.965560    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.965686 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.965628    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.967595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.967544    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:44.967792 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.967769    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.969056 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.969034    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:44.989595 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:44.989482    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:45.013767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.009579    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9gxx\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:45.013767 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.011474    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:45.015321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.015283    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" event={"ID":"91ed7422-66b4-4198-9b85-956064954579","Type":"ContainerStarted","Data":"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"}
Nov 02 12:53:45.015484 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.015380    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:45.027754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.027723    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m25t\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:45.074408 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.074373    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:45.084392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.084336    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" podStartSLOduration=2.703052969 podStartE2EDuration="3.084317659s" totalImagesPullingTime="381.26469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:45.080155129 +0000 UTC m=+1622.729209086" watchObservedRunningTime="2026-11-02 12:53:45.084317659 +0000 UTC m=+1622.733371618"
Nov 02 12:53:45.085905 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.085866    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:45.428476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.428446    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:53:45.429775 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:45.429750    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fde714_9cb2_49a3_96a4_420d84e51552.slice/crio-babb751653eacb226f45aeb3b3938ad9d720dd85fd9f256a70eaa30907d7481e WatchSource:0}: Error finding container babb751653eacb226f45aeb3b3938ad9d720dd85fd9f256a70eaa30907d7481e: Status 404 returned error can't find the container with id babb751653eacb226f45aeb3b3938ad9d720dd85fd9f256a70eaa30907d7481e
Nov 02 12:53:45.433173 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.433131    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.433265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.433222    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.433304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.433269    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.443051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.443026    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:53:45.445982 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:45.445958    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc51ab2_86b3_4b7a_abc1_4542cb27576b.slice/crio-959315a06192e8828b2cf22b55a0da08fb86f2efe68b9c461c52f5188b4006aa WatchSource:0}: Error finding container 959315a06192e8828b2cf22b55a0da08fb86f2efe68b9c461c52f5188b4006aa: Status 404 returned error can't find the container with id 959315a06192e8828b2cf22b55a0da08fb86f2efe68b9c461c52f5188b4006aa
Nov 02 12:53:45.448531 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.448502    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.448603 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.448577    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.448642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.448624    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:53:45.912539 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:45.912504    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 12:53:46.122906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.122817    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" event={"ID":"17fde714-9cb2-49a3-96a4-420d84e51552","Type":"ContainerStarted","Data":"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"}
Nov 02 12:53:46.123393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.122937    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" event={"ID":"17fde714-9cb2-49a3-96a4-420d84e51552","Type":"ContainerStarted","Data":"babb751653eacb226f45aeb3b3938ad9d720dd85fd9f256a70eaa30907d7481e"}
Nov 02 12:53:46.195758 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.195631    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" event={"ID":"4cc51ab2-86b3-4b7a-abc1-4542cb27576b","Type":"ContainerStarted","Data":"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"}
Nov 02 12:53:46.200272 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.200204    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" event={"ID":"4cc51ab2-86b3-4b7a-abc1-4542cb27576b","Type":"ContainerStarted","Data":"959315a06192e8828b2cf22b55a0da08fb86f2efe68b9c461c52f5188b4006aa"}
Nov 02 12:53:46.200529 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.198760    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podStartSLOduration=2.198742667 podStartE2EDuration="2.198742667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:46.19717501 +0000 UTC m=+1623.846228969" watchObservedRunningTime="2026-11-02 12:53:46.198742667 +0000 UTC m=+1623.847796628"
Nov 02 12:53:46.921343 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.921305    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 12:53:46.993791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:46.993732    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podStartSLOduration=2.993714704 podStartE2EDuration="2.993714704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:46.30097945 +0000 UTC m=+1623.950033408" watchObservedRunningTime="2026-11-02 12:53:46.993714704 +0000 UTC m=+1624.642768661"
Nov 02 12:53:47.088632 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.088597    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:47.089233 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.089210    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:47.090401 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.090360    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:47.090584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.090557    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:47.091447 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.091425    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:47.091616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:47.091591    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:48.076600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:48.076562    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:48.077474 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:48.077440    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:48.086225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:48.086202    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:48.086927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:48.086893    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:49.078286 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.078244    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:49.078882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.078846    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:49.086537 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.086513    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:49.086715 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.086693    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:49.456904 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.454979    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 12:53:49.494852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.493761    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 12:53:49.494852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.494691    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.510304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.510266    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.510515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.510494    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.510651 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.510632    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psf6m\" (UniqueName: \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.510759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.510742    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.514265 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.514239    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e7561a-serving-cert\""
Nov 02 12:53:49.616063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.616024    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psf6m\" (UniqueName: \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.616686 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.616535    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.616813 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:53:49.616710    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-serving-cert: secret "maas-api-e2e-shared-a-w3-e7561a-serving-cert" not found
Nov 02 12:53:49.616813 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:53:49.616782    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls podName:39df5dfc-af28-4dcb-be2c-0e01a12c9519 nodeName:}" failed. No retries permitted until 2026-11-02 12:53:50.11676142 +0000 UTC m=+1627.765815361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" (UID: "39df5dfc-af28-4dcb-be2c-0e01a12c9519") : secret "maas-api-e2e-shared-a-w3-e7561a-serving-cert" not found
Nov 02 12:53:49.616963 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.616850    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.617348 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.617322    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.618017 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.617984    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.623838 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.623794    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:49.687864 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:49.687787    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psf6m\" (UniqueName: \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:50.103848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.091137    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:50.103848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.091315    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:50.103848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.091606    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:50.103848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.091738    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:50.160852 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.160261    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:50.235847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.227506    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:50.498085 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.497128    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:50.894939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:50.894358    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 12:53:51.079751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.079280    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:51.079751 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.079348    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:51.112161 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.112110    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:51.112813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.112738    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:51.205154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.205083    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 12:53:51.521872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:51.521806    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" event={"ID":"39df5dfc-af28-4dcb-be2c-0e01a12c9519","Type":"ContainerStarted","Data":"4fc0946fd62f33da2d137306acbcad54d975f740e4c63c179113729ae088cb12"}
Nov 02 12:53:52.074997 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.074957    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:52.075243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.075026    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 12:53:52.086840 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.086792    4928 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 12:53:52.087013 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.086877    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 12:53:52.532804 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.532765    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" event={"ID":"39df5dfc-af28-4dcb-be2c-0e01a12c9519","Type":"ContainerStarted","Data":"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"}
Nov 02 12:53:52.533326 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.532877    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:52.584257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:52.584197    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" podStartSLOduration=3.194603216 podStartE2EDuration="3.584182765s" totalImagesPullingTime="389.579549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:52.583938106 +0000 UTC m=+1630.232992065" watchObservedRunningTime="2026-11-02 12:53:52.584182765 +0000 UTC m=+1630.233236723"
Nov 02 12:53:53.257981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.257947    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:53.260507 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.260472    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:53.261757 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.261737    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:53:53.335359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.335326    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:53.335725 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.335698    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:53.336979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:53.336953    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:53:57.040840 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.040797    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 12:53:57.047608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.047582    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.073300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.073268    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8032f3\""
Nov 02 12:53:57.073494 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.073467    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 12:53:57.073890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.073863    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8032f3-dockercfg-lf775\""
Nov 02 12:53:57.075207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.075185    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.075330 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.075256    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8hp\" (UniqueName: \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.180044 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.180001    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.318242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.318132    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8hp\" (UniqueName: \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.318596 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.181650    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.345223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.345189    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8hp\" (UniqueName: \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\") pod \"payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.366442 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.366373    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:57.719430 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:57.719392    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 12:53:57.720774 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:53:57.720742    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf4f19c_d345_426e_8f70_20843296706d.slice/crio-1998fbf7e4d50d0e61768eed73a99297f829f2ed46a31df96316fcee8b392bc8 WatchSource:0}: Error finding container 1998fbf7e4d50d0e61768eed73a99297f829f2ed46a31df96316fcee8b392bc8: Status 404 returned error can't find the container with id 1998fbf7e4d50d0e61768eed73a99297f829f2ed46a31df96316fcee8b392bc8
Nov 02 12:53:58.552214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:58.552185    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 12:53:58.600321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:58.600280    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" event={"ID":"3cf4f19c-d345-426e-8f70-20843296706d","Type":"ContainerStarted","Data":"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"}
Nov 02 12:53:58.600584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:58.600562    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" event={"ID":"3cf4f19c-d345-426e-8f70-20843296706d","Type":"ContainerStarted","Data":"1998fbf7e4d50d0e61768eed73a99297f829f2ed46a31df96316fcee8b392bc8"}
Nov 02 12:53:58.602780 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:58.602745    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:53:58.673945 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:53:58.673884    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" podStartSLOduration=1.67386721 podStartE2EDuration="1.67386721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:58.671138948 +0000 UTC m=+1636.320192909" watchObservedRunningTime="2026-11-02 12:53:58.67386721 +0000 UTC m=+1636.322921193"
Nov 02 12:54:05.082879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.082841    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:54:05.087517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.087494    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.118956 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.118920    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:54:05.124667 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.124615    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-74ba71-openshift-default-dockercfg-pkjws\""
Nov 02 12:54:05.194276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194221    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.194569 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194536    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.194733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194662    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx8m\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.194733 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194701    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.194894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194773    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.194894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194880    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.195019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194926    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.195019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194960    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.195019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.194988    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296483    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296523 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296530    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296549    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296597    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296656    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296755    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx8m\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.296813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296789    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296872    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.296967    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297108 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.297059    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.297106    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297335 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.297301    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.297352    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.297453 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.297407    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.299252 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.299228    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.299378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.299356    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.325440 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.325403    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx8m\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.334818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.334718    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\") pod \"e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") " pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.403251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.403212    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:05.756918 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:54:05.756879    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ac4b8e_e9d8_4627_9c96_eeced1b18db2.slice/crio-799a4042103ebf6d4396a6dea110da5cf81c565f4a48b427b79d7b70a8e7ac01 WatchSource:0}: Error finding container 799a4042103ebf6d4396a6dea110da5cf81c565f4a48b427b79d7b70a8e7ac01: Status 404 returned error can't find the container with id 799a4042103ebf6d4396a6dea110da5cf81c565f4a48b427b79d7b70a8e7ac01
Nov 02 12:54:05.758457 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.758426    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:54:05.760136 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.760093    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:05.760220 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.760182    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:05.760260 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:05.760225    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:06.726646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:06.726593    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerStarted","Data":"f7087be93e539f51cfc23b98f989c7853a08ca40c1e2a34253952606cdcb0c34"}
Nov 02 12:54:06.727380 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:06.727353    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerStarted","Data":"799a4042103ebf6d4396a6dea110da5cf81c565f4a48b427b79d7b70a8e7ac01"}
Nov 02 12:54:06.800683 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:06.800613    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podStartSLOduration=1.8005927480000001 podStartE2EDuration="1.800592748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:06.796496345 +0000 UTC m=+1644.445550302" watchObservedRunningTime="2026-11-02 12:54:06.800592748 +0000 UTC m=+1644.449646706"
Nov 02 12:54:07.404070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:07.404031    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:07.410516 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:07.410491    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:07.733854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:07.733788    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:07.734884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:07.734860    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:09.614376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:09.614343    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 12:54:09.735473 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:54:09.735405    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:54:11.290367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.290325    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:54:11.295934 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.295908    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.316139 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.316117    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-74ba71\""
Nov 02 12:54:11.324290 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.324265    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.324401 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.324311    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxkm\" (UniqueName: \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.329753 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.329718    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-74ba71-dockercfg-dcld2\""
Nov 02 12:54:11.334771 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.334743    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:54:11.446851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.445699    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.446851 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.445761    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxkm\" (UniqueName: \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.447097 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.447052    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.561509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.561398    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxkm\" (UniqueName: \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\") pod \"payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.611652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.611611    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:11.930787 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:11.930748    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:54:11.932067 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:54:11.932039    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9835ae_8ccd_4904_bd69_abb8e36d52dc.slice/crio-3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9 WatchSource:0}: Error finding container 3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9: Status 404 returned error can't find the container with id 3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9
Nov 02 12:54:12.859899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:12.846696    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" event={"ID":"3e9835ae-8ccd-4904-bd69-abb8e36d52dc","Type":"ContainerStarted","Data":"a143890251d8ff18dd841e913d3f5ebe21d5cf0b1d4dbaeb4939cd0324dbe818"}
Nov 02 12:54:12.859899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:12.846806    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" event={"ID":"3e9835ae-8ccd-4904-bd69-abb8e36d52dc","Type":"ContainerStarted","Data":"3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9"}
Nov 02 12:54:12.859899 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:12.850859    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:14.747437 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.747363    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" podStartSLOduration=3.747342485 podStartE2EDuration="3.747342485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:12.920106936 +0000 UTC m=+1650.569160895" watchObservedRunningTime="2026-11-02 12:54:14.747342485 +0000 UTC m=+1652.396396444"
Nov 02 12:54:14.752276 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.752242    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 12:54:14.758049 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.758021    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.764126 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.764100    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 12:54:14.781135 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.781105    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-1e14d4-openshift-default-dockercfg-9n6bp\""
Nov 02 12:54:14.821614 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821578    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821620    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821645    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821783 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821729    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821859    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821914 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821881    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzxv\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.821989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.821962    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.822032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.822024    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.822124 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.822109    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.930807    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.930878    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzxv\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.930955    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931036    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931137    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931232    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931266    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931305    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.931854 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.931362    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.932502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.932316    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.935224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.935192    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.947848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.936484    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.947848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.936726    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.948422 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.948397    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.957038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.957007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:14.991181 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:14.991141    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:15.003939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.000752    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:15.016991 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.009027    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzxv\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:15.083202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.082861    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:15.433196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.431150    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 12:54:15.435503 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:54:15.435468    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29253340_c10e_49c2_9b68_d63a92f59d84.slice/crio-dbd70fb0144e96db4751485170d78becbbc1511af21ad9a285c355aca9ae5d1c WatchSource:0}: Error finding container dbd70fb0144e96db4751485170d78becbbc1511af21ad9a285c355aca9ae5d1c: Status 404 returned error can't find the container with id dbd70fb0144e96db4751485170d78becbbc1511af21ad9a285c355aca9ae5d1c
Nov 02 12:54:15.439810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.439223    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:15.439810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.439322    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:15.439810 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.439368    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:15.869345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.869309    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerStarted","Data":"d6d305585add06a6128fec036d7d83a5c608a4e4290657b465f1bc0a174ee1fa"}
Nov 02 12:54:15.869754 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:15.869395    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerStarted","Data":"dbd70fb0144e96db4751485170d78becbbc1511af21ad9a285c355aca9ae5d1c"}
Nov 02 12:54:16.084146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.084109    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:16.090069 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.090029    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:16.161322 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.161191    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podStartSLOduration=2.161171609 podStartE2EDuration="2.161171609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:15.971771107 +0000 UTC m=+1653.620825065" watchObservedRunningTime="2026-11-02 12:54:16.161171609 +0000 UTC m=+1653.810225568"
Nov 02 12:54:16.718678 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.718644    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 12:54:16.724411 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.724363    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.746304 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.746270    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6416f0-openshift-default-dockercfg-bkn6v\""
Nov 02 12:54:16.756673 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.756642    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886359    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886421    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886487    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886642    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886674    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886704    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886737    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2852h\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886781    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.886848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.886839    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:16.924855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.923874    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:16.984159 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.982636    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.987964    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988017    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988047    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988074    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2852h\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988113    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988150    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988253    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988291    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988378    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.988858    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.998961    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.999225    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:16.999850    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.014755 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.000110    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.018927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.017630    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.060746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.056335    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2852h\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.080849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.068436    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.080849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.077852    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\") pod \"e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") " pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.354962 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.348338    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:17.741214 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.741184    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 12:54:17.741496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.741459    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:17.741650 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.741555    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:17.741650 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.741617    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:17.933765 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.933726    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerStarted","Data":"37a2d5d63a6f63314960794fac2d631c5ce454117616c66d72f13d3f3fe29ee1"}
Nov 02 12:54:17.934241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:17.933796    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerStarted","Data":"29d54a95fa4d2904a91b2bed938195cac1f3f6e954b1f93876158978f3bc84d6"}
Nov 02 12:54:18.007419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:18.007297    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podStartSLOduration=2.007278673 podStartE2EDuration="2.007278673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:18.006463957 +0000 UTC m=+1655.655517909" watchObservedRunningTime="2026-11-02 12:54:18.007278673 +0000 UTC m=+1655.656332631"
Nov 02 12:54:18.349548 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:18.349451    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:18.351269 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:18.351235    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:54:18.351412 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:18.351298    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:54:19.354740 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:19.354703    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:19.973642 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:19.973608    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:19.974627 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:19.974599    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:54:24.881395 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:24.880115    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:54:37.449197 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.449157    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 12:54:37.453916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.453871    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.499946 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.495660    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-1aa18c-serving-cert\""
Nov 02 12:54:37.502730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.501562    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 12:54:37.576041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.575998    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.576227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.576056    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.576227 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.576183    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.576369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.576353    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4mr\" (UniqueName: \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.682543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.679085    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4mr\" (UniqueName: \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.682543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.679245    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.682543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.679292    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.682543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.679373    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.687120 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.686209    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.696716 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.691265    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.725943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.725857    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.731807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.731775    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4mr\" (UniqueName: \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\") pod \"maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:37.769955 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:37.769916    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:38.089164 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:38.089129    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 12:54:38.090181 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:54:38.090154    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdc46b5_2b8e_467f_b888_b269c3bbba6f.slice/crio-e3be77491727509efff06f5b6aded42e7b5d81f6f1db1dc7eaafc1ee9b78e120 WatchSource:0}: Error finding container e3be77491727509efff06f5b6aded42e7b5d81f6f1db1dc7eaafc1ee9b78e120: Status 404 returned error can't find the container with id e3be77491727509efff06f5b6aded42e7b5d81f6f1db1dc7eaafc1ee9b78e120
Nov 02 12:54:38.159449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:38.159404    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" event={"ID":"1bdc46b5-2b8e-467f-b888-b269c3bbba6f","Type":"ContainerStarted","Data":"e3be77491727509efff06f5b6aded42e7b5d81f6f1db1dc7eaafc1ee9b78e120"}
Nov 02 12:54:39.172530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:39.172496    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" event={"ID":"1bdc46b5-2b8e-467f-b888-b269c3bbba6f","Type":"ContainerStarted","Data":"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"}
Nov 02 12:54:39.173040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:39.172599    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:39.221243 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:39.221191    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" podStartSLOduration=1.73430211 podStartE2EDuration="2.221176792s" totalImagesPullingTime="486.874682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:39.218288917 +0000 UTC m=+1676.867342857" watchObservedRunningTime="2026-11-02 12:54:39.221176792 +0000 UTC m=+1676.870230749"
Nov 02 12:54:39.457521 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:39.457427    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:54:39.457727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:39.457670    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" containerID="cri-o://90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221" gracePeriod=30
Nov 02 12:54:39.730589 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:54:39.730500    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:54:40.288369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:40.288326    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2x5c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:54:40.288775 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:40.288385    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:54:44.785321 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.785292    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:54:44.917047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.916965    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917047 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917011    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917090    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917112    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917199    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917289    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917346    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917522 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917385    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.917670 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917531    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\") pod \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\" (UID: \"533bf8a6-2c2e-4b9c-923f-6096f276a9ee\") "
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917377    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.917759    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918016    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918029    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918107    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918223    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918237    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918245    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918254    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.918346 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.918262    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.919809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.919785    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:54:44.919923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.919881    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:54:44.920497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.920468    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.920497 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:44.920489    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv" pod "533bf8a6-2c2e-4b9c-923f-6096f276a9ee" (UID: "533bf8a6-2c2e-4b9c-923f-6096f276a9ee"). InnerVolumeSpecName "kube-api-access-dkkjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:54:45.019443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.019403    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkjv\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-kube-api-access-dkkjv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.019443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.019437    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.019443 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.019450    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.019759 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.019463    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533bf8a6-2c2e-4b9c-923f-6096f276a9ee-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.218461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.217205    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 12:54:45.235892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.231197    4928 generic.go:356] "Generic (PLEG): container finished" podID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerID="90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221" exitCode=0
Nov 02 12:54:45.235892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.231279    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerDied","Data":"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"}
Nov 02 12:54:45.235892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.231326    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4" event={"ID":"533bf8a6-2c2e-4b9c-923f-6096f276a9ee","Type":"ContainerDied","Data":"54af0422d0c7fc75c535a900b3a79f8664442cc29921897eef6529d40ffaa627"}
Nov 02 12:54:45.235892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.231358    4928 scope.go:122] "RemoveContainer" containerID="90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"
Nov 02 12:54:45.235892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.231536    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"
Nov 02 12:54:45.249041 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.249019    4928 scope.go:122] "RemoveContainer" containerID="fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"
Nov 02 12:54:45.259338 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.259307    4928 scope.go:122] "RemoveContainer" containerID="90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"
Nov 02 12:54:45.259614 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:54:45.259594    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221\": container with ID starting with 90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221 not found: ID does not exist" containerID="90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"
Nov 02 12:54:45.259707 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.259622    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221"} err="failed to get container status \"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221\": rpc error: code = NotFound desc = could not find container \"90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221\": container with ID starting with 90dda17bfa3f27c1176dfe723f98e52e156928891d388c5734aa89806c6b4221 not found: ID does not exist"
Nov 02 12:54:45.259707 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.259639    4928 scope.go:122] "RemoveContainer" containerID="fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"
Nov 02 12:54:45.259945 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:54:45.259927    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f\": container with ID starting with fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f not found: ID does not exist" containerID="fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"
Nov 02 12:54:45.260007 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.259962    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f"} err="failed to get container status \"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f\": rpc error: code = NotFound desc = could not find container \"fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f\": container with ID starting with fb476822f8b4396b0b5979ad892bea9f6310d02836509a99395cf6a9f543431f not found: ID does not exist"
Nov 02 12:54:45.391860 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.391240    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:54:45.420848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:45.419980    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2x5c4"]
Nov 02 12:54:46.498868 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.498810    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" path="/var/lib/kubelet/pods/533bf8a6-2c2e-4b9c-923f-6096f276a9ee/volumes"
Nov 02 12:54:46.664588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.664536    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 12:54:46.665882 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.665854    4928 cpu_manager.go:414] "containerMap: removing container" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:46.666045 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.666029    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:46.666185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.666172    4928 cpu_manager.go:414] "containerMap: removing container" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:46.666280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.666267    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:46.666584 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.666565    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:46.672661 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.672642    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.710848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.707204    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 12:54:46.724502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.724465    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-43be82-openshift-default-dockercfg-4zxkp\""
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774586    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774645    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkkq\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774682    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774770    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774816    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774863    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774937    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.774968    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.781937 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.775054    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.884790    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885021    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885056    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkkq\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885089    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885177    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885219    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885251    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885329    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885363    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.894848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.885928    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.955024 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.954986    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.955556 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.955521    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.956139 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.956108    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.957271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.957237    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.962809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.960440    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.962809 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.961818    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:46.996742 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:46.996700    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:47.046900 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.037732    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkkq\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\") pod \"e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") " pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:47.306213 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.306122    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:47.489845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.479974    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 12:54:47.489845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.481259    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="533bf8a6-2c2e-4b9c-923f-6096f276a9ee" containerName="istio-proxy"
Nov 02 12:54:47.489845 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.489036    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.503019 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.498751    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 12:54:47.510402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.510377    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6416f0-serving-cert\""
Nov 02 12:54:47.578055 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.577687    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.578055 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.577892    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.578055 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.577969    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648mj\" (UniqueName: \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.578368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.578060    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.681240 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.681193    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648mj\" (UniqueName: \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.687981 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.687941    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.689977 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.689947    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.690663 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.690643    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.700075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.692568    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.700075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.697271    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.711119 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.709757    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.713052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.712609    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-648mj\" (UniqueName: \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\") pod \"maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:47.718890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.715700    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:47.718890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.715795    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:47.718890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.715855    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:47.721691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.720104    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 12:54:47.812493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:47.812460    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:48.178891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:48.177887    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 12:54:48.308205 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:48.308164    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerStarted","Data":"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"}
Nov 02 12:54:48.308544 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:48.308519    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerStarted","Data":"2e8d8f35d7d37916894cb8554ea77358f2b2837f28f10ed3f16eb1acdaf60feb"}
Nov 02 12:54:48.309665 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:48.309640    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" event={"ID":"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7","Type":"ContainerStarted","Data":"cefa77d09f0c3b91b1c087adadb726907530ee1a2934c8d8717eb2fe55469e2a"}
Nov 02 12:54:48.384878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:48.381908    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podStartSLOduration=2.381886045 podStartE2EDuration="2.381886045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:48.373873887 +0000 UTC m=+1686.022927846" watchObservedRunningTime="2026-11-02 12:54:48.381886045 +0000 UTC m=+1686.030940004"
Nov 02 12:54:49.306932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.306897    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:49.314639 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.314613    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:49.319814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.319788    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" event={"ID":"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7","Type":"ContainerStarted","Data":"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"}
Nov 02 12:54:49.319948 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.319879    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:49.320098 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.320080    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:49.320927 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.320909    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:54:49.445849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:49.441325    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" podStartSLOduration=1.851719647 podStartE2EDuration="2.441301152s" totalImagesPullingTime="589.581505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:49.424079344 +0000 UTC m=+1687.073133302" watchObservedRunningTime="2026-11-02 12:54:49.441301152 +0000 UTC m=+1687.090355111"
Nov 02 12:54:53.777850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:53.777126    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:54:53.777850 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:53.777221    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:55.348400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.348329    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 12:54:55.392572 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.392537    4928 generic.go:356] "Generic (PLEG): container finished" podID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerID="f7087be93e539f51cfc23b98f989c7853a08ca40c1e2a34253952606cdcb0c34" exitCode=0
Nov 02 12:54:55.392734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.392622    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerDied","Data":"f7087be93e539f51cfc23b98f989c7853a08ca40c1e2a34253952606cdcb0c34"}
Nov 02 12:54:55.393170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.393151    4928 scope.go:122] "RemoveContainer" containerID="f7087be93e539f51cfc23b98f989c7853a08ca40c1e2a34253952606cdcb0c34"
Nov 02 12:54:55.395324 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.395291    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:55.395463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.395379    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:55.395463 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:55.395429    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:54:56.402442 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:56.402407    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerStarted","Data":"98b2a31445ee95050f39f813dfc6097f3df62183c0b37916a8d30a9a02537ef8"}
Nov 02 12:54:56.403870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:56.403817    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:54:56.404075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:56.404053    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:54:56.404154 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:56.404104    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:54:57.403553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:57.403517    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:54:57.404079 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:57.403585    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:54:58.403704 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:58.403666    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:54:58.404141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:58.403735    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:54:59.403621 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:59.403578    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:54:59.403883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:54:59.403668    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:55:00.403640 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:00.403606    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:55:00.403815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:00.403675    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:55:01.404559 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:01.404518    4928 patch_prober.go:29] interesting pod/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 12:55:01.405067 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:01.404599    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 12:55:02.747084 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:02.747047    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:55:02.747617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:02.747586    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:55:02.748547 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:02.748523    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:55:09.764345 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:55:09.764302    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:55:17.966339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:17.965976    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:55:17.966339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:17.966058    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:55:20.771711 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.771680    4928 generic.go:356] "Generic (PLEG): container finished" podID="29253340-c10e-49c2-9b68-d63a92f59d84" containerID="d6d305585add06a6128fec036d7d83a5c608a4e4290657b465f1bc0a174ee1fa" exitCode=0
Nov 02 12:55:20.773088 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.771843    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerDied","Data":"d6d305585add06a6128fec036d7d83a5c608a4e4290657b465f1bc0a174ee1fa"}
Nov 02 12:55:20.773088 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.772321    4928 scope.go:122] "RemoveContainer" containerID="d6d305585add06a6128fec036d7d83a5c608a4e4290657b465f1bc0a174ee1fa"
Nov 02 12:55:20.775414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.774931    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:20.775414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.775019    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:20.775414 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.775067    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:20.901374 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.901333    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": read tcp 10.132.0.2:60138->10.132.0.161:15021: read: connection reset by peer" start-of-body=
Nov 02 12:55:20.901509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:20.901443    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": read tcp 10.132.0.2:60138->10.132.0.161:15021: read: connection reset by peer"
Nov 02 12:55:21.784408 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.784369    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerStarted","Data":"137e15d7a6ae5598355109f8acf448800cbdfb8ca04b702bf78096d0fde1f6bd"}
Nov 02 12:55:21.786105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.786075    4928 generic.go:356] "Generic (PLEG): container finished" podID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerID="37a2d5d63a6f63314960794fac2d631c5ce454117616c66d72f13d3f3fe29ee1" exitCode=0
Nov 02 12:55:21.786256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.786144    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerDied","Data":"37a2d5d63a6f63314960794fac2d631c5ce454117616c66d72f13d3f3fe29ee1"}
Nov 02 12:55:21.786585 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.786566    4928 scope.go:122] "RemoveContainer" containerID="37a2d5d63a6f63314960794fac2d631c5ce454117616c66d72f13d3f3fe29ee1"
Nov 02 12:55:21.788288 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.788256    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:21.788362 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.788331    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:21.788400 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:21.788366    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:55:22.084040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:22.083945    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:22.084040 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:22.084006    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:22.084273 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:22.084186    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:55:22.807939 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:22.807895    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerStarted","Data":"eb8e5a6aca16c7a94cceef09e9a7b486519c02602d5f3d4870cf1ea19c52c518"}
Nov 02 12:55:23.088622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:23.088351    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:23.088622 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:23.088417    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:23.360916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:23.360228    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:55:23.360916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:23.360394    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:55:23.360916 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:23.360441    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:55:24.091132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:24.089388    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:24.091132 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:24.089448    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:24.349349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:24.349248    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:55:24.349669 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:24.349627    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:55:25.093853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:25.090114    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:25.093853 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:25.090198    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:25.374253 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:25.374158    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:55:25.374515 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:25.374483    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:55:26.094287 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:26.094228    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:26.094909 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:26.094871    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:26.352301 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:26.352202    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:55:26.352574 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:26.352545    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:55:27.084261 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:27.084222    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:55:27.084461 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:27.084278    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 12:55:27.356712 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:27.356611    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:55:27.357167 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:27.356704    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 12:55:28.089061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.089024    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:55:28.089482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.089461    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:55:28.090369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.090347    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 12:55:28.363450 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.363330    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:55:28.365419 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.365390    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:55:28.366649 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:55:28.366628    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 12:55:39.732315 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:55:39.732267    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:56:01.217747 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.217711    4928 generic.go:356] "Generic (PLEG): container finished" podID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerID="e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780" exitCode=0
Nov 02 12:56:01.218247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.217783    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerDied","Data":"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"}
Nov 02 12:56:01.218301 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.218285    4928 scope.go:122] "RemoveContainer" containerID="e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"
Nov 02 12:56:01.220300 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.220268    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:56:01.220433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.220340    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:56:01.220433 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:01.220378    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:56:02.264870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:02.264810    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerStarted","Data":"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"}
Nov 02 12:56:02.307216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:02.307178    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:56:02.308883 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:02.308821    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:02.309023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:02.308904    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:03.308067 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:03.308030    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:03.308500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:03.308084    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:04.307103 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:04.307062    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:04.307290 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:04.307128    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:04.320349 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:04.320320    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:56:05.307582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:05.307543    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:05.307772 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:05.307600    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:06.307890 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:06.307852    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:06.308403 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:06.307922    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:07.307737 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:07.307699    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 12:56:07.307932 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:07.307760    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 12:56:08.311884 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:08.311819    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:56:08.313316 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:08.313290    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 12:56:09.730574 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:56:09.730534    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:56:24.457081 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.457043    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 12:56:24.464099 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.464065    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.485231 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.485202    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 12:56:24.490089 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.490060    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6416f0\"/\"kube-root-ca.crt\""
Nov 02 12:56:24.490242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.490133    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6416f0\"/\"openshift-service-ca.crt\""
Nov 02 12:56:24.490242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.490232    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6416f0\"/\"default-dockercfg-b7854\""
Nov 02 12:56:24.494425 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.494390    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6416f0\"/\"auth-test-model-6416f0-kserve-self-signed-certs\""
Nov 02 12:56:24.537533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.537504    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.537718 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.537625    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.537797 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.537770    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.537974 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.537953    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrckf\" (UniqueName: \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.538112 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.538096    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.538302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.538283    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.639886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.639801    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.639905    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640006    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrckf\" (UniqueName: \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640090 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640174    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640252    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640382 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640360    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640460    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.640654 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.640624    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.642530 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.642509    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.642902 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.642883    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.668155 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.668095    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrckf\" (UniqueName: \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\") pod \"auth-test-model-6416f0-kserve-74f98db469-djqph\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") " pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:24.788855 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:24.787915    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:25.117202 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:25.117172    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 12:56:25.118319 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:56:25.118289    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6720dc5a_0e0a_44e0_b108_b0ef78768eea.slice/crio-d93ee9ec7bbd941f7b8aac3b0e73562ce01058edff8f1ba98bd53c6b8fcbd452 WatchSource:0}: Error finding container d93ee9ec7bbd941f7b8aac3b0e73562ce01058edff8f1ba98bd53c6b8fcbd452: Status 404 returned error can't find the container with id d93ee9ec7bbd941f7b8aac3b0e73562ce01058edff8f1ba98bd53c6b8fcbd452
Nov 02 12:56:25.473685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:25.473583    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" event={"ID":"6720dc5a-0e0a-44e0-b108-b0ef78768eea","Type":"ContainerStarted","Data":"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"}
Nov 02 12:56:25.473685 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:25.473646    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" event={"ID":"6720dc5a-0e0a-44e0-b108-b0ef78768eea","Type":"ContainerStarted","Data":"d93ee9ec7bbd941f7b8aac3b0e73562ce01058edff8f1ba98bd53c6b8fcbd452"}
Nov 02 12:56:25.519402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:25.519353    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" podStartSLOduration=1.519339765 podStartE2EDuration="1.519339765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:56:25.516954194 +0000 UTC m=+1783.166008156" watchObservedRunningTime="2026-11-02 12:56:25.519339765 +0000 UTC m=+1783.168394085"
Nov 02 12:56:25.790923 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:25.790887    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:34.790921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:34.790886    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:34.809128 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:34.809100    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:35.568445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:35.568408    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 12:56:39.730225 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:56:39.730187    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:56:42.876514 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:42.876479    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:56:42.877746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:56:42.877725    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 12:57:09.122971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.122931    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 12:57:09.127396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.127370    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.145312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.144375    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1aa18c-openshift-default-dockercfg-wk4h7\""
Nov 02 12:57:09.179402 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.179151    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 12:57:09.303938 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.303889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304214    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304394 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304279    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304394 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304331    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304520    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304616 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304598    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304680 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304666    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304741 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304692    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlpp\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.304807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.304736    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406377 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406286    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406377 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406367    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406417    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406617 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406549    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlpp\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406734 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406621    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406738    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406777    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406843    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.406924 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.406907    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.407177 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.407147    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.407359 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.407324    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.407528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.407502    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.407528 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.407517    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.407634 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.407569    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.408994 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.408972    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.409618 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.409599    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.431003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.430975    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlpp\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.431221 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.431189    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.447399 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.447375    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:09.731582 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:57:09.731496    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:57:09.780364 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.780334    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 12:57:09.781150 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:57:09.781107    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c018ca_6757_48f1_836b_02cadf9f088f.slice/crio-203d5ae52a5b154f0f343eb85b72568933adf965d0f69bc9383c9a534ac73a90 WatchSource:0}: Error finding container 203d5ae52a5b154f0f343eb85b72568933adf965d0f69bc9383c9a534ac73a90: Status 404 returned error can't find the container with id 203d5ae52a5b154f0f343eb85b72568933adf965d0f69bc9383c9a534ac73a90
Nov 02 12:57:09.784351 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.784308    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:57:09.784483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.784397    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:57:09.784483 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.784450    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:57:09.957722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.957613    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" event={"ID":"24c018ca-6757-48f1-836b-02cadf9f088f","Type":"ContainerStarted","Data":"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"}
Nov 02 12:57:09.957722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:09.957677    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" event={"ID":"24c018ca-6757-48f1-836b-02cadf9f088f","Type":"ContainerStarted","Data":"203d5ae52a5b154f0f343eb85b72568933adf965d0f69bc9383c9a534ac73a90"}
Nov 02 12:57:10.007271 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:10.007216    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podStartSLOduration=1.007201945 podStartE2EDuration="1.007201945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:57:10.004524127 +0000 UTC m=+1827.653578085" watchObservedRunningTime="2026-11-02 12:57:10.007201945 +0000 UTC m=+1827.656255902"
Nov 02 12:57:10.454917 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:10.453510    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:10.458222 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:10.458179    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:10.458390 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:10.458246    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:11.452848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:11.448948    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:11.452848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:11.449012    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:12.455856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:12.452237    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:12.455856 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:12.452371    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:13.449848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:13.449266    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:13.449848 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:13.449350    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:14.450280 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:14.450230    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:14.450879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:14.450314    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:15.450849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:15.450184    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:57:15.450849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:15.450250    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 02 12:57:16.454109 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:16.454069    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:16.454559 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:16.454463    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:16.455298 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:16.455274    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 12:57:39.731707 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:57:39.731661    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:57:40.084509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:40.084480    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:57:45.087921 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.085518    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" podUID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" containerName="payload-pre-processing" containerID="cri-o://a143890251d8ff18dd841e913d3f5ebe21d5cf0b1d4dbaeb4939cd0324dbe818" gracePeriod=25
Nov 02 12:57:45.377458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.377426    4928 generic.go:356] "Generic (PLEG): container finished" podID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" containerID="a143890251d8ff18dd841e913d3f5ebe21d5cf0b1d4dbaeb4939cd0324dbe818" exitCode=0
Nov 02 12:57:45.377652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.377508    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" event={"ID":"3e9835ae-8ccd-4904-bd69-abb8e36d52dc","Type":"ContainerDied","Data":"a143890251d8ff18dd841e913d3f5ebe21d5cf0b1d4dbaeb4939cd0324dbe818"}
Nov 02 12:57:45.377652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.377585    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t" event={"ID":"3e9835ae-8ccd-4904-bd69-abb8e36d52dc","Type":"ContainerDied","Data":"3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9"}
Nov 02 12:57:45.377652 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.377611    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b5d292b240ff1c07f7ce317e573458bd76983e6aedf2ff1af566e314d40e8f9"
Nov 02 12:57:45.389955 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.389933    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:57:45.459628 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.459596    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\") pod \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") "
Nov 02 12:57:45.459818 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.459737    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\" (UniqueName: \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\") pod \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\" (UID: \"3e9835ae-8ccd-4904-bd69-abb8e36d52dc\") "
Nov 02 12:57:45.461074 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.461037    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume" pod "3e9835ae-8ccd-4904-bd69-abb8e36d52dc" (UID: "3e9835ae-8ccd-4904-bd69-abb8e36d52dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:45.462563 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.462532    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm" pod "3e9835ae-8ccd-4904-bd69-abb8e36d52dc" (UID: "3e9835ae-8ccd-4904-bd69-abb8e36d52dc"). InnerVolumeSpecName "kube-api-access-6zxkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:45.561958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.561920    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:57:45.561958 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:45.561959    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxkm\" (UniqueName: \"kubernetes.io/projected/3e9835ae-8ccd-4904-bd69-abb8e36d52dc-kube-api-access-6zxkm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:57:46.385800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:46.385757    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"
Nov 02 12:57:46.442777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:46.442745    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:57:46.456464 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:46.456427    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-74ba71-766cbbf4bc-5m29t"]
Nov 02 12:57:46.499249 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:46.499210    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" path="/var/lib/kubelet/pods/3e9835ae-8ccd-4904-bd69-abb8e36d52dc/volumes"
Nov 02 12:57:55.031588 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:55.031550    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"]
Nov 02 12:57:55.044282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:55.044252    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad426528-4zrw5"]
Nov 02 12:57:56.499152 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:57:56.499105    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da124b66-675c-4740-9f8a-bfa5b02e3d40" path="/var/lib/kubelet/pods/da124b66-675c-4740-9f8a-bfa5b02e3d40/volumes"
Nov 02 12:58:09.730729 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:58:09.730685    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:58:11.609062 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:11.609026    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:58:11.609610 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:11.609343    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy" containerID="cri-o://98b2a31445ee95050f39f813dfc6097f3df62183c0b37916a8d30a9a02537ef8" gracePeriod=30
Nov 02 12:58:15.032116 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:15.032077    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"]
Nov 02 12:58:15.047028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:15.046995    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b263cf5-dn7hs"]
Nov 02 12:58:16.499137 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.499099    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff6110e-a0c8-456e-9ac3-40a200310572" path="/var/lib/kubelet/pods/2ff6110e-a0c8-456e-9ac3-40a200310572/volumes"
Nov 02 12:58:16.772815 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.772696    4928 generic.go:356] "Generic (PLEG): container finished" podID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerID="98b2a31445ee95050f39f813dfc6097f3df62183c0b37916a8d30a9a02537ef8" exitCode=0
Nov 02 12:58:16.773376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.772914    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerDied","Data":"98b2a31445ee95050f39f813dfc6097f3df62183c0b37916a8d30a9a02537ef8"}
Nov 02 12:58:16.773376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.772987    4928 scope.go:122] "RemoveContainer" containerID="f7087be93e539f51cfc23b98f989c7853a08ca40c1e2a34253952606cdcb0c34"
Nov 02 12:58:16.926493 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.926461    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:58:16.968872 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.968810    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.968968    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969037    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969077 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969066    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969104    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969150    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969159    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:16.969244 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969211    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969277    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969352    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\") pod \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\" (UID: \"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2\") "
Nov 02 12:58:16.969553 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969522    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:16.969730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969684    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:16.970020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.969976    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:16.971557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971182    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:16.971557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971213    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:16.971557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971228    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:16.971557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971243    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:16.971557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971354    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:16.971912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.971886    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "kube-api-access-6vx8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:16.972345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.972322    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:16.972507 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.972479    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:16.972571 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:16.972502    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy" pod "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" (UID: "b2ac4b8e-e9d8-4627-9c96-eeced1b18db2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:17.072798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.072711    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vx8m\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-kube-api-access-6vx8m\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:17.072798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.072742    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:17.072798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.072752    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:17.072798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.072761    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:17.072798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.072769    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:58:17.782629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.782593    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"
Nov 02 12:58:17.782629 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.782616    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44" event={"ID":"b2ac4b8e-e9d8-4627-9c96-eeced1b18db2","Type":"ContainerDied","Data":"799a4042103ebf6d4396a6dea110da5cf81c565f4a48b427b79d7b70a8e7ac01"}
Nov 02 12:58:17.783169 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.782684    4928 scope.go:122] "RemoveContainer" containerID="98b2a31445ee95050f39f813dfc6097f3df62183c0b37916a8d30a9a02537ef8"
Nov 02 12:58:17.855862 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.855813    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:58:17.870582 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:17.870542    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-74ba71-openshift-default-d75495477-bbs44"]
Nov 02 12:58:18.499078 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:18.499043    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" path="/var/lib/kubelet/pods/b2ac4b8e-e9d8-4627-9c96-eeced1b18db2/volumes"
Nov 02 12:58:39.445543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.445503    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 12:58:39.446557 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446531    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" containerName="payload-pre-processing"
Nov 02 12:58:39.446613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446560    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" containerName="payload-pre-processing"
Nov 02 12:58:39.446613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446596    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446613 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446604    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446637    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446645    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446870    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446895    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ac4b8e-e9d8-4627-9c96-eeced1b18db2" containerName="istio-proxy"
Nov 02 12:58:39.446943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.446917    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9835ae-8ccd-4904-bd69-abb8e36d52dc" containerName="payload-pre-processing"
Nov 02 12:58:39.451367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.451346    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.473656 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.473622    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 12:58:39.480068 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.480044    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a6f2c0-openshift-default-dockercfg-7djh5\""
Nov 02 12:58:39.558627 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.558585    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.558858 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.558695    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4kt\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559247 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559215    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559460    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559578 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559507    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559743 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559718    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559889 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559859    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559906    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.559975 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.559962    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.661885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.661843    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.661885 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.661896    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.661950    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662017    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662146 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662082    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4kt\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662180    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662307    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662312    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662380    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662420    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662679    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662729 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662716    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.662971 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.662945    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.663372 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.663349    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.664798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.664777    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.665242 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.665221    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.688603 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.687638    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.690849 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.690444    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4kt\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:39.731092 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:58:39.731006    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:58:39.767075 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:39.767041    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:40.097722 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.097688    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 12:58:40.098725 ip-10-0-135-160 kubenswrapper[4928]: W1102 12:58:40.098699    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbee72f1_e429_4ca4_bbf5_076da013775f.slice/crio-511c31b4722ca46aaef869716c261d08cfad0899a503c6859c3d07aa8dd8f65c WatchSource:0}: Error finding container 511c31b4722ca46aaef869716c261d08cfad0899a503c6859c3d07aa8dd8f65c: Status 404 returned error can't find the container with id 511c31b4722ca46aaef869716c261d08cfad0899a503c6859c3d07aa8dd8f65c
Nov 02 12:58:40.101517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.101479    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:58:40.101611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.101548    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:58:40.101611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.101585    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 12:58:40.993003 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.992960    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" event={"ID":"dbee72f1-e429-4ca4-bbf5-076da013775f","Type":"ContainerStarted","Data":"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"}
Nov 02 12:58:40.993509 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:40.993035    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" event={"ID":"dbee72f1-e429-4ca4-bbf5-076da013775f","Type":"ContainerStarted","Data":"511c31b4722ca46aaef869716c261d08cfad0899a503c6859c3d07aa8dd8f65c"}
Nov 02 12:58:41.073873 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:41.073787    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podStartSLOduration=2.073763752 podStartE2EDuration="2.073763752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:41.055906992 +0000 UTC m=+1918.704960950" watchObservedRunningTime="2026-11-02 12:58:41.073763752 +0000 UTC m=+1918.722817711"
Nov 02 12:58:41.768166 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:41.768119    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:41.770384 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:41.770355    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:41.770555 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:41.770437    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:42.768037 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:42.767995    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:42.768469 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:42.768073    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:43.767625 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:43.767585    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:43.767886 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:43.767677    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:44.768123 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:44.768084    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:44.768543 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:44.768154    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:45.767838 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:45.767796    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:45.768034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:45.767870    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:46.768301 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:46.768254    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:58:46.768727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:46.768349    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 12:58:47.772813 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:47.772780    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:47.773435 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:47.773273    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:58:47.774070 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:58:47.774045    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 12:59:09.734098 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:09.734016    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:59:30.733708 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.733673    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:59:30.734393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.734020    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy" containerID="cri-o://f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f" gracePeriod=30
Nov 02 12:59:30.747517 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.747475    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:59:30.747769 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.747736    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy" containerID="cri-o://ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a" gracePeriod=30
Nov 02 12:59:30.751991 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.751966    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:59:30.752179 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.752143    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy" containerID="cri-o://c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc" gracePeriod=30
Nov 02 12:59:30.760023 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.759998    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:59:30.760200 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:30.760179    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy" containerID="cri-o://1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0" gracePeriod=30
Nov 02 12:59:31.406730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:31.406690    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:59:31.406933 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:31.406908    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy" containerID="cri-o://13ec57970e57aa2c422c70c897d432a4a3853792278ca6a80607b4861fe53752" gracePeriod=30
Nov 02 12:59:34.327611 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:34.327557    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:59:34.328105 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:34.327845    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy" containerID="cri-o://f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a" gracePeriod=30
Nov 02 12:59:36.070857 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.070805    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:59:36.155328 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.155293    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.155906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.155881    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.156111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.156093    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.156295 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.155797    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.156365 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.156264    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.156441 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.156287    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.156647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.156632    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.157133 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.157088    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.157378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.156942    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.157695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.157442    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.157695 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.157665    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.157912 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.157332    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.158072 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.158056    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.159274 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.159255    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\") pod \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\" (UID: \"ffcb3c32-3642-45c5-b99c-54497f8bcf40\") "
Nov 02 12:59:36.160502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.160479    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.160635 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.160622    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.160724 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.160713    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.160807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.160797    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.160989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.160976    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.171774 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.171734    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.171952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.171795    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "kube-api-access-wx22j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.172061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.172030    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.173337 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.173312    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy" pod "ffcb3c32-3642-45c5-b99c-54497f8bcf40" (UID: "ffcb3c32-3642-45c5-b99c-54497f8bcf40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.262207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.262158    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.262207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.262196    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.262207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.262209    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.262207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.262219    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx22j\" (UniqueName: \"kubernetes.io/projected/ffcb3c32-3642-45c5-b99c-54497f8bcf40-kube-api-access-wx22j\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.528618 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.528584    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:59:36.593586 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.593526    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:59:36.597870 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.595437    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:59:36.639897 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.639859    4928 generic.go:356] "Generic (PLEG): container finished" podID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerID="ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a" exitCode=0
Nov 02 12:59:36.640357 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.640333    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"
Nov 02 12:59:36.640526 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.640328    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" event={"ID":"4cc51ab2-86b3-4b7a-abc1-4542cb27576b","Type":"ContainerDied","Data":"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"}
Nov 02 12:59:36.640620 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.640602    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc" event={"ID":"4cc51ab2-86b3-4b7a-abc1-4542cb27576b","Type":"ContainerDied","Data":"959315a06192e8828b2cf22b55a0da08fb86f2efe68b9c461c52f5188b4006aa"}
Nov 02 12:59:36.640688 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.640648    4928 scope.go:122] "RemoveContainer" containerID="ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.651389    4928 generic.go:356] "Generic (PLEG): container finished" podID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerID="f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f" exitCode=0
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.651477    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" event={"ID":"e4895448-e75c-4614-8bbd-39b2538bb7a9","Type":"ContainerDied","Data":"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.651529    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p" event={"ID":"e4895448-e75c-4614-8bbd-39b2538bb7a9","Type":"ContainerDied","Data":"24a40d496c60d3e75e37cd394a46ca1ad2febb9a0cbd814bea8342113c937edc"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.651611    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.654309    4928 generic.go:356] "Generic (PLEG): container finished" podID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerID="13ec57970e57aa2c422c70c897d432a4a3853792278ca6a80607b4861fe53752" exitCode=0
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.654440    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" event={"ID":"3808189d-ad8a-429c-8207-4b09f8ad9ce7","Type":"ContainerDied","Data":"13ec57970e57aa2c422c70c897d432a4a3853792278ca6a80607b4861fe53752"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.656281    4928 generic.go:356] "Generic (PLEG): container finished" podID="17fde714-9cb2-49a3-96a4-420d84e51552" containerID="c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc" exitCode=0
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.656387    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" event={"ID":"17fde714-9cb2-49a3-96a4-420d84e51552","Type":"ContainerDied","Data":"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.656421    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h" event={"ID":"17fde714-9cb2-49a3-96a4-420d84e51552","Type":"ContainerDied","Data":"babb751653eacb226f45aeb3b3938ad9d720dd85fd9f256a70eaa30907d7481e"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.656495    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.659454    4928 generic.go:356] "Generic (PLEG): container finished" podID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerID="1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0" exitCode=0
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.659514    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" event={"ID":"ffcb3c32-3642-45c5-b99c-54497f8bcf40","Type":"ContainerDied","Data":"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.659553    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft" event={"ID":"ffcb3c32-3642-45c5-b99c-54497f8bcf40","Type":"ContainerDied","Data":"086905c53771f10921b0ceb21847a89a1fb8130585dfadd87e1c9ee746e167e2"}
Nov 02 12:59:36.663066 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.659621    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"
Nov 02 12:59:36.669684 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.669589    4928 scope.go:122] "RemoveContainer" containerID="ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"
Nov 02 12:59:36.669977 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:36.669950    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a\": container with ID starting with ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a not found: ID does not exist" containerID="ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"
Nov 02 12:59:36.670076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.669988    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a"} err="failed to get container status \"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a\": rpc error: code = NotFound desc = could not find container \"ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a\": container with ID starting with ef1ac8eb83502c139af2519dc332028026ec1ad53919dd319f689a9e094dd72a not found: ID does not exist"
Nov 02 12:59:36.670076 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.670011    4928 scope.go:122] "RemoveContainer" containerID="f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"
Nov 02 12:59:36.685424 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685337    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.685598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685424    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.685598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685461    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.685598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685504    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.685598 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685550    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685600    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685642    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685672    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685723    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685778    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.685847 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685806    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.686170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.685950    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.686170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686015    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.686170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686063    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.686170 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686100    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.686378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686206    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.686378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686256    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.686378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686295    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.686378 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686340    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.686691 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686659    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.686952 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.686927    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687756    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687801    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687799    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687861    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687906    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\") pod \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\" (UID: \"4cc51ab2-86b3-4b7a-abc1-4542cb27576b\") "
Nov 02 12:59:36.688009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.687972    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.688814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.688055    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.688814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.688113    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\") pod \"e4895448-e75c-4614-8bbd-39b2538bb7a9\" (UID: \"e4895448-e75c-4614-8bbd-39b2538bb7a9\") "
Nov 02 12:59:36.688814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.688156    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\") pod \"17fde714-9cb2-49a3-96a4-420d84e51552\" (UID: \"17fde714-9cb2-49a3-96a4-420d84e51552\") "
Nov 02 12:59:36.690257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.689490    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "kube-api-access-jxh9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.690257 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.690226    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.694892 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.694846    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.696028 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.696001    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.697472 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.696240    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.697472 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.696511    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.697472 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.696788    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.697730 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.697512    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.697979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.697959    4928 scope.go:122] "RemoveContainer" containerID="f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"
Nov 02 12:59:36.698431 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.698226    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.698533 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.698430    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.698600 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.698589    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.698721 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.698706    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.698985 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:36.698958    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f\": container with ID starting with f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f not found: ID does not exist" containerID="f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"
Nov 02 12:59:36.699224 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699192    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f"} err="failed to get container status \"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f\": rpc error: code = NotFound desc = could not find container \"f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f\": container with ID starting with f2bf6e255d8967b3a17535b209cd6e9feb4c9aac13b060bdaea3c22ea47fd71f not found: ID does not exist"
Nov 02 12:59:36.699347 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699334    4928 scope.go:122] "RemoveContainer" containerID="c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"
Nov 02 12:59:36.699498 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699011    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.699915 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699436    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.700038 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699715    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.700151 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.699496    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.700268 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.700253    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.700368 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.700357    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.701345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.701234    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.702790 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.702769    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.703393 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.703373    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.706199 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.706173    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo" pod "e4895448-e75c-4614-8bbd-39b2538bb7a9" (UID: "e4895448-e75c-4614-8bbd-39b2538bb7a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.710545 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.710524    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "kube-api-access-7m25t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.710687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.710560    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.710800 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.710742    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "kube-api-access-q9gxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.712954 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.712920    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo" pod "17fde714-9cb2-49a3-96a4-420d84e51552" (UID: "17fde714-9cb2-49a3-96a4-420d84e51552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.713050 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.712955    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo" pod "4cc51ab2-86b3-4b7a-abc1-4542cb27576b" (UID: "4cc51ab2-86b3-4b7a-abc1-4542cb27576b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.719053 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.718723    4928 scope.go:122] "RemoveContainer" containerID="c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"
Nov 02 12:59:36.719367 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:36.719229    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc\": container with ID starting with c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc not found: ID does not exist" containerID="c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"
Nov 02 12:59:36.719367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.719267    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc"} err="failed to get container status \"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc\": rpc error: code = NotFound desc = could not find container \"c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc\": container with ID starting with c874f7ae3e9a36a085e0d23ba565d2bd2727ae2cc2e54cc7961a5a9b5832ebfc not found: ID does not exist"
Nov 02 12:59:36.719367 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.719292    4928 scope.go:122] "RemoveContainer" containerID="1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"
Nov 02 12:59:36.735251 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.735224    4928 scope.go:122] "RemoveContainer" containerID="1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"
Nov 02 12:59:36.735584 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:36.735556    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0\": container with ID starting with 1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0 not found: ID does not exist" containerID="1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"
Nov 02 12:59:36.735677 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.735593    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0"} err="failed to get container status \"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0\": rpc error: code = NotFound desc = could not find container \"1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0\": container with ID starting with 1fc7eb5e11326c96a83d28f13c8cffb84ee9ddc53543380044af4540bc093bf0 not found: ID does not exist"
Nov 02 12:59:36.794878 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.794733    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:59:36.806061 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806020    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-kzbft"]
Nov 02 12:59:36.806312 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806171    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4895448-e75c-4614-8bbd-39b2538bb7a9-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806319    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806337    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806348    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9gxx\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-kube-api-access-q9gxx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806358    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m25t\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-kube-api-access-7m25t\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806367    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806375    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806384    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806391    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806400    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806408    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806417    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806425    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxh9l\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-kube-api-access-jxh9l\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806432    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde714-9cb2-49a3-96a4-420d84e51552-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806441    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde714-9cb2-49a3-96a4-420d84e51552-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806449    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806457    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806465    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde714-9cb2-49a3-96a4-420d84e51552-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806472    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde714-9cb2-49a3-96a4-420d84e51552-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806480    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806489    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.806478 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806497    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4895448-e75c-4614-8bbd-39b2538bb7a9-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.807687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806504    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.807687 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.806512    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc51ab2-86b3-4b7a-abc1-4542cb27576b-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.830396 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:36.830359    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:59:37.002030 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.001992    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:59:37.006445 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.006420    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-9v8fc"]
Nov 02 12:59:37.008647 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008622    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.008746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008701    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.008746 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008740    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.008841 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008772    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.008907 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008853    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.009009 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.008969    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.009064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009041    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.009064 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009055    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:37.009217 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009201    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.009282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009263    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:37.009282 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009264    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:37.009394 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009294    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\") pod \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\" (UID: \"3808189d-ad8a-429c-8207-4b09f8ad9ce7\") "
Nov 02 12:59:37.009458 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009423    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009640    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009960    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.009984    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.010000    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.010013    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.010051 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.010026    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.011381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.011358    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:37.011646 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.011625    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:37.011745 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.011660    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:37.013980 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.013959    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp" pod "3808189d-ad8a-429c-8207-4b09f8ad9ce7" (UID: "3808189d-ad8a-429c-8207-4b09f8ad9ce7"). InnerVolumeSpecName "kube-api-access-plgtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:37.044989 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.044914    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:59:37.053471 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.053438    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-62f6p"]
Nov 02 12:59:37.113093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.113018    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:59:37.114080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.114049    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plgtp\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-kube-api-access-plgtp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.114080 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.114078    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.114225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.114090    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.114225 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.114102    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3808189d-ad8a-429c-8207-4b09f8ad9ce7-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.118992 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.118966    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-xqj7h"]
Nov 02 12:59:37.675979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.675949    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"
Nov 02 12:59:37.675979 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.675965    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx" event={"ID":"3808189d-ad8a-429c-8207-4b09f8ad9ce7","Type":"ContainerDied","Data":"ad2f1a09f8289f1b5160888f8935bf7fc462f8f763b33e63468ae5a911b3c031"}
Nov 02 12:59:37.676258 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.676042    4928 scope.go:122] "RemoveContainer" containerID="13ec57970e57aa2c422c70c897d432a4a3853792278ca6a80607b4861fe53752"
Nov 02 12:59:37.734504 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.734469    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:59:37.739481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:37.739453    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-jvkjx"]
Nov 02 12:59:38.215345 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.215310    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:59:38.215772 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.215558    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy" containerID="cri-o://b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa" gracePeriod=30
Nov 02 12:59:38.219791 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.219747    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:59:38.220020 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.219996    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy" containerID="cri-o://2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08" gracePeriod=30
Nov 02 12:59:38.499805 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.499774    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" path="/var/lib/kubelet/pods/17fde714-9cb2-49a3-96a4-420d84e51552/volumes"
Nov 02 12:59:38.500363 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.500346    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" path="/var/lib/kubelet/pods/3808189d-ad8a-429c-8207-4b09f8ad9ce7/volumes"
Nov 02 12:59:38.500807 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.500794    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" path="/var/lib/kubelet/pods/4cc51ab2-86b3-4b7a-abc1-4542cb27576b/volumes"
Nov 02 12:59:38.501630 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.501614    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" path="/var/lib/kubelet/pods/e4895448-e75c-4614-8bbd-39b2538bb7a9/volumes"
Nov 02 12:59:38.502093 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:38.502078    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" path="/var/lib/kubelet/pods/ffcb3c32-3642-45c5-b99c-54497f8bcf40/volumes"
Nov 02 12:59:39.650589 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.650559    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:59:39.654666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.654642    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.654801 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.654783    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.654894 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.654874    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.654955 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.654934    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655032 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655015    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655106 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655048    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655106 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655079    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655172    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655185    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.655223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655198    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.655223 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655212    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\") pod \"baaa51fb-4f4e-43d8-8f77-696f3067b111\" (UID: \"baaa51fb-4f4e-43d8-8f77-696f3067b111\") "
Nov 02 12:59:39.655499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655303    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:39.655499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655307    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.655499 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655430    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.655891 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.655866    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baaa51fb-4f4e-43d8-8f77-696f3067b111-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.656034 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.656020    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.656157 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.656134    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.656275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.656159    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.656275 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.656175    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.657500 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.657478    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "kube-api-access-s8ctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:39.657608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.657516    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.657608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.657551    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:39.657608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.657595    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token" pod "baaa51fb-4f4e-43d8-8f77-696f3067b111" (UID: "baaa51fb-4f4e-43d8-8f77-696f3067b111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:39.706145 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.706108    4928 generic.go:356] "Generic (PLEG): container finished" podID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerID="f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a" exitCode=0
Nov 02 12:59:39.706310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.706176    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"
Nov 02 12:59:39.706310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.706192    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" event={"ID":"baaa51fb-4f4e-43d8-8f77-696f3067b111","Type":"ContainerDied","Data":"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"}
Nov 02 12:59:39.706310 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.706286    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487" event={"ID":"baaa51fb-4f4e-43d8-8f77-696f3067b111","Type":"ContainerDied","Data":"5a1c6e8eb81ac2eb85bf31c172fd0dd5b40c30ae0704cd3a85e39473d6169252"}
Nov 02 12:59:39.706573 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.706314    4928 scope.go:122] "RemoveContainer" containerID="f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"
Nov 02 12:59:39.727185 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.727043    4928 scope.go:122] "RemoveContainer" containerID="f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"
Nov 02 12:59:39.727486 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:39.727459    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a\": container with ID starting with f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a not found: ID does not exist" containerID="f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"
Nov 02 12:59:39.727579 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.727494    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a"} err="failed to get container status \"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a\": rpc error: code = NotFound desc = could not find container \"f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a\": container with ID starting with f988b2bf0df1d05ff1e22516b47b2200e762898180288d71212d885ffbfc906a not found: ID does not exist"
Nov 02 12:59:39.748703 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:39.748666    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 12:59:39.757798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.757756    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.757798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.757789    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.757798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.757798    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baaa51fb-4f4e-43d8-8f77-696f3067b111-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.757798 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.757809    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ctp\" (UniqueName: \"kubernetes.io/projected/baaa51fb-4f4e-43d8-8f77-696f3067b111-kube-api-access-s8ctp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.780392 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.780357    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:59:39.783814 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:39.783779    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-4b487"]
Nov 02 12:59:40.386538 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:40.386500    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:59:40.386750 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:40.386713    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" containerID="cri-o://0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01" gracePeriod=30
Nov 02 12:59:40.507626 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:40.507590    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" path="/var/lib/kubelet/pods/baaa51fb-4f4e-43d8-8f77-696f3067b111/volumes"
Nov 02 12:59:43.539706 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.539333    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:59:43.576256 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.576231    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633350    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633503    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633581    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633635    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633666    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633669    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633701    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.633764 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633737    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633803    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633884    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633918    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.633981    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634008    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634080    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634108    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634118    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634133    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634245    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\") pod \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\" (UID: \"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634334    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634345    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634368    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:43.634482 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634421    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\") pod \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\" (UID: \"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad\") "
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634589    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.634790    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635188    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635239    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635264    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635279    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635268    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635294    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635308    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.635381 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635321    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.636012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635623    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.636012 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.635896    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.640984 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.640956    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.641117 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.641003    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.642974    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.642999    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d" pod "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" (UID: "e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad"). InnerVolumeSpecName "kube-api-access-8s69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.643029    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.643043    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.643061    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "kube-api-access-pdtnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:43.650871 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.643084    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token" pod "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" (UID: "38b0c9f4-8de9-47fe-bfc5-06f70082f4c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:43.737194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737159    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737191    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737200    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s69d\" (UniqueName: \"kubernetes.io/projected/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-kube-api-access-8s69d\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737194 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737211    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737220    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737228    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737236    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737244    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737251    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737259    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737266    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.737489 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.737276    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtnx\" (UniqueName: \"kubernetes.io/projected/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0-kube-api-access-pdtnx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.746481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.746396    4928 generic.go:356] "Generic (PLEG): container finished" podID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerID="2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08" exitCode=0
Nov 02 12:59:43.746481 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.746453    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"
Nov 02 12:59:43.746666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.746486    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" event={"ID":"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0","Type":"ContainerDied","Data":"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"}
Nov 02 12:59:43.746666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.746547    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7" event={"ID":"38b0c9f4-8de9-47fe-bfc5-06f70082f4c0","Type":"ContainerDied","Data":"2fc88f3346cb2a2893469bbb5974250db056c36ec3df774fccae25af69ab22aa"}
Nov 02 12:59:43.746666 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.746570    4928 scope.go:122] "RemoveContainer" containerID="2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"
Nov 02 12:59:43.749052 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.749022    4928 generic.go:356] "Generic (PLEG): container finished" podID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerID="b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa" exitCode=0
Nov 02 12:59:43.749196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.749106    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" event={"ID":"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad","Type":"ContainerDied","Data":"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"}
Nov 02 12:59:43.749196 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.749142    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr" event={"ID":"e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad","Type":"ContainerDied","Data":"8faa09686237445f14257396201245b24e0b8299b271e5ecaac3746bcbf7a321"}
Nov 02 12:59:43.749376 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.749200    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"
Nov 02 12:59:43.761753 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.761733    4928 scope.go:122] "RemoveContainer" containerID="2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"
Nov 02 12:59:43.762122 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:43.762095    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08\": container with ID starting with 2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08 not found: ID does not exist" containerID="2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"
Nov 02 12:59:43.762216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.762130    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08"} err="failed to get container status \"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08\": rpc error: code = NotFound desc = could not find container \"2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08\": container with ID starting with 2f4bc803fc2bd3f7e260e917ac6692ff78481ae4a97f0d48228124346f0d6b08 not found: ID does not exist"
Nov 02 12:59:43.762216 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.762152    4928 scope.go:122] "RemoveContainer" containerID="b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"
Nov 02 12:59:43.776876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.776193    4928 scope.go:122] "RemoveContainer" containerID="b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"
Nov 02 12:59:43.776876 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:43.776726    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa\": container with ID starting with b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa not found: ID does not exist" containerID="b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"
Nov 02 12:59:43.776876 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.776788    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa"} err="failed to get container status \"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa\": rpc error: code = NotFound desc = could not find container \"b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa\": container with ID starting with b4386a51890466c7490b45e3a798c645c64ac828061b72050025f0b19e2f78aa not found: ID does not exist"
Nov 02 12:59:43.786906 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.786869    4928 scope.go:122] "RemoveContainer" containerID="49fabcb042624d7dc1718ecaf8f5e98ee4271e20f432a9e5dbeea2ca760a76f5"
Nov 02 12:59:43.799777 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.799742    4928 scope.go:122] "RemoveContainer" containerID="c250e366d64355015830aa21e3ec74473e5e0b965b93bd3975d9dbf040e8beaa"
Nov 02 12:59:43.806460 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.806432    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:59:43.813608 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.813578    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-pnsp7"]
Nov 02 12:59:43.867496 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.867467    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:59:43.875255 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:43.875232    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-6xvhr"]
Nov 02 12:59:44.499535 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:44.499454    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" path="/var/lib/kubelet/pods/38b0c9f4-8de9-47fe-bfc5-06f70082f4c0/volumes"
Nov 02 12:59:44.500005 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:44.499980    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" path="/var/lib/kubelet/pods/e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad/volumes"
Nov 02 12:59:44.862541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:44.862449    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zdk72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:59:44.862541 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:44.862512    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:59:45.665341 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.665272    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:59:45.766835 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.766782    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767033 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.766959    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767119    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767198 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767160    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767262 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767201    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767315 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767261    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767391 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767375    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767449 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767440    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767502 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767492    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\") pod \"993299a0-d111-4e9b-b2be-ae6e54b1a032\" (UID: \"993299a0-d111-4e9b-b2be-ae6e54b1a032\") "
Nov 02 12:59:45.767943 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767915    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.768063 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.767945    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:45.768141 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768121    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.768241 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768223    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993299a0-d111-4e9b-b2be-ae6e54b1a032-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.768302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768249    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.768302 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768263    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.768407 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768320    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.768476 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.768454    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.772229 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.772201    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.775111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.774240    4928 generic.go:356] "Generic (PLEG): container finished" podID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerID="0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01" exitCode=0
Nov 02 12:59:45.775111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.774456    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" event={"ID":"993299a0-d111-4e9b-b2be-ae6e54b1a032","Type":"ContainerDied","Data":"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"}
Nov 02 12:59:45.775111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.774509    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72" event={"ID":"993299a0-d111-4e9b-b2be-ae6e54b1a032","Type":"ContainerDied","Data":"91ef55c063548e8a9f0e45a8e6d238d2729cbaa401f25d5953e8c693bfd32116"}
Nov 02 12:59:45.775111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.774540    4928 scope.go:122] "RemoveContainer" containerID="0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"
Nov 02 12:59:45.775111 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.774656    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"
Nov 02 12:59:45.777339 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.777307    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:45.777693 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.777669    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:45.777781 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.777670    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5" pod "993299a0-d111-4e9b-b2be-ae6e54b1a032" (UID: "993299a0-d111-4e9b-b2be-ae6e54b1a032"). InnerVolumeSpecName "kube-api-access-gfwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:45.805294 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.805261    4928 scope.go:122] "RemoveContainer" containerID="0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"
Nov 02 12:59:45.805673 ip-10-0-135-160 kubenswrapper[4928]: E1102 12:59:45.805645    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01\": container with ID starting with 0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01 not found: ID does not exist" containerID="0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"
Nov 02 12:59:45.805727 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.805681    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01"} err="failed to get container status \"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01\": rpc error: code = NotFound desc = could not find container \"0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01\": container with ID starting with 0383e99a6e55869bbcbd39d6fbb7046da43e271995d9b541f305ab923ffaea01 not found: ID does not exist"
Nov 02 12:59:45.869369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869330    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfwm5\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-kube-api-access-gfwm5\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.869369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869364    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.869369 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869376    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.869879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869387    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.869879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869395    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.869879 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:45.869404    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993299a0-d111-4e9b-b2be-ae6e54b1a032-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 12:59:46.129207 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:46.129166    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:59:46.143987 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:46.143947    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zdk72"]
Nov 02 12:59:46.498946 ip-10-0-135-160 kubenswrapper[4928]: I1102 12:59:46.498911    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" path="/var/lib/kubelet/pods/993299a0-d111-4e9b-b2be-ae6e54b1a032/volumes"
Nov 02 13:00:00.151402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.151359    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:00.152245 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152221    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy"
Nov 02 13:00:00.152245 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152245    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152265    4928 cpu_manager.go:414] "containerMap: removing container" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152273    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152285    4928 cpu_manager.go:414] "containerMap: removing container" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152293    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152313    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152321    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152346    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152353    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152368    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152376    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152390    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152398    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152418    4928 cpu_manager.go:414] "containerMap: removing container" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy"
Nov 02 13:00:00.152442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152428    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152454    4928 cpu_manager.go:414] "containerMap: removing container" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152464    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152618    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="993299a0-d111-4e9b-b2be-ae6e54b1a032" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152635    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3808189d-ad8a-429c-8207-4b09f8ad9ce7" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152648    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc51ab2-86b3-4b7a-abc1-4542cb27576b" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152658    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4895448-e75c-4614-8bbd-39b2538bb7a9" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152669    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcb3c32-3642-45c5-b99c-54497f8bcf40" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152679    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fde714-9cb2-49a3-96a4-420d84e51552" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152690    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b0c9f4-8de9-47fe-bfc5-06f70082f4c0" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152702    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61699b6-c8e7-41a4-bbd9-7cbbf8a490ad" containerName="istio-proxy"
Nov 02 13:00:00.153122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.152712    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="baaa51fb-4f4e-43d8-8f77-696f3067b111" containerName="istio-proxy"
Nov 02 13:00:00.158492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.158469    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.179478 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.177858    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:00.182811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.182779    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.182944 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.182785    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:00.187860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.187582    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:00.187860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.187668    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.189352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.189324    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:00:00.208875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.208809    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:00.211530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.211494    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.211713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.211686    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mm8\" (UniqueName: \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.232122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.232092    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:00.232349 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.232336    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:00.232691 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.232562    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.261376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.261345    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:00.313334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313048    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.313334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313208    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.313334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313319    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.313714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313375    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mm8\" (UniqueName: \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.313714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313406    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92h4k\" (UniqueName: \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.313714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313483    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jwf\" (UniqueName: \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.313714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313542    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfbs\" (UniqueName: \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.313714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.313674    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.314448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.314426    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.338058 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.338024    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mm8\" (UniqueName: \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.414889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.414775    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.414889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.414879    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92h4k\" (UniqueName: \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.415115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.414926    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jwf\" (UniqueName: \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.415115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.414964    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfbs\" (UniqueName: \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.415115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.415040    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.415115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.415072    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.415547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.415526    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.415910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.415880    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.416000 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.415887    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.440538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.440483    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92h4k\" (UniqueName: \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.440751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.440728    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jwf\" (UniqueName: \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.441426 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.441402    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfbs\" (UniqueName: \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\") pod \"maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.481254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.481230    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:00.509556 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.509525    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:00.533151 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.533116    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:00.565754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.559131    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:00.799874 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.799840    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:00.800847 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:00.800796    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153356b1_810d_4a44_ad73_e28bbd69fdec.slice/crio-e4f0c065307509cd2d30ff0a0bd8c8e5c823ea76560883bb85147f8da447ee7a WatchSource:0}: Error finding container e4f0c065307509cd2d30ff0a0bd8c8e5c823ea76560883bb85147f8da447ee7a: Status 404 returned error can't find the container with id e4f0c065307509cd2d30ff0a0bd8c8e5c823ea76560883bb85147f8da447ee7a
Nov 02 13:00:00.854983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.854812    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:00.861335 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:00.861283    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b68dec3_73cb_4d8a_aaf5_d929a03726db.slice/crio-f99db303376ef5ab4afac693cbfabd9512b9e63af81e31a34f2d67c55d4fe45d WatchSource:0}: Error finding container f99db303376ef5ab4afac693cbfabd9512b9e63af81e31a34f2d67c55d4fe45d: Status 404 returned error can't find the container with id f99db303376ef5ab4afac693cbfabd9512b9e63af81e31a34f2d67c55d4fe45d
Nov 02 13:00:00.892398 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.892276    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:00.902544 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.899170    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerStarted","Data":"f99db303376ef5ab4afac693cbfabd9512b9e63af81e31a34f2d67c55d4fe45d"}
Nov 02 13:00:00.903237 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.903205    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerStarted","Data":"e4f0c065307509cd2d30ff0a0bd8c8e5c823ea76560883bb85147f8da447ee7a"}
Nov 02 13:00:00.922160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:00.921999    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:00.931077 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:00.931046    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c8245e_f6b7_4aed_92d4_50bb39450819.slice/crio-72d2b769dac3daad25df1e5b2c7a56ffca7aaee96521f8c97f442decdf8f758f WatchSource:0}: Error finding container 72d2b769dac3daad25df1e5b2c7a56ffca7aaee96521f8c97f442decdf8f758f: Status 404 returned error can't find the container with id 72d2b769dac3daad25df1e5b2c7a56ffca7aaee96521f8c97f442decdf8f758f
Nov 02 13:00:01.913918 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.913787    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerStarted","Data":"3102d496a2e2060750d2b9f723736993967c4b50c1e1c1961c3ed9df818a6150"}
Nov 02 13:00:01.913918 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.913860    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerStarted","Data":"72d2b769dac3daad25df1e5b2c7a56ffca7aaee96521f8c97f442decdf8f758f"}
Nov 02 13:00:01.915560 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.915534    4928 generic.go:356] "Generic (PLEG): container finished" podID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerID="c0593fe2f931ae16743a4d715459acd64a52b3729e1aaaca26a52a0133953646" exitCode=60
Nov 02 13:00:01.915690 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.915612    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerDied","Data":"c0593fe2f931ae16743a4d715459acd64a52b3729e1aaaca26a52a0133953646"}
Nov 02 13:00:01.915892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.915874    4928 scope.go:122] "RemoveContainer" containerID="c0593fe2f931ae16743a4d715459acd64a52b3729e1aaaca26a52a0133953646"
Nov 02 13:00:01.917302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.917282    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerStarted","Data":"851c9cc4d4a6f6fc8f991eed3a19789667eb405fdb6704abf32561ebf9e88c1e"}
Nov 02 13:00:01.919044 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.919023    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerStarted","Data":"f41a4e0f75556ed2f764e74cf2b42c95f90e4c6b90551c005b6f55fc14c708b4"}
Nov 02 13:00:01.919135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.919064    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerStarted","Data":"8a530c64b0fc443421f971127f4c1cf2aed60400e1a2f74e267839b2f41c4334"}
Nov 02 13:00:01.955953 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.955903    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" podStartSLOduration=1.955888591 podStartE2EDuration="1.955888591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.955006347 +0000 UTC m=+1999.604060306" watchObservedRunningTime="2026-11-02 13:00:01.955888591 +0000 UTC m=+1999.604942548"
Nov 02 13:00:01.997310 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:01.997252    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" podStartSLOduration=1.997235666 podStartE2EDuration="1.997235666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.997153263 +0000 UTC m=+1999.646207222" watchObservedRunningTime="2026-11-02 13:00:01.997235666 +0000 UTC m=+1999.646289625"
Nov 02 13:00:02.086117 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.086063    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" podStartSLOduration=2.086049156 podStartE2EDuration="2.086049156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:02.08194944 +0000 UTC m=+1999.731003401" watchObservedRunningTime="2026-11-02 13:00:02.086049156 +0000 UTC m=+1999.735103119"
Nov 02 13:00:02.934153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.934109    4928 generic.go:356] "Generic (PLEG): container finished" podID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerID="3102d496a2e2060750d2b9f723736993967c4b50c1e1c1961c3ed9df818a6150" exitCode=60
Nov 02 13:00:02.935110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.935076    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerDied","Data":"3102d496a2e2060750d2b9f723736993967c4b50c1e1c1961c3ed9df818a6150"}
Nov 02 13:00:02.935551 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.935530    4928 scope.go:122] "RemoveContainer" containerID="3102d496a2e2060750d2b9f723736993967c4b50c1e1c1961c3ed9df818a6150"
Nov 02 13:00:02.939348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.939291    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerStarted","Data":"4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"}
Nov 02 13:00:02.942666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.942633    4928 generic.go:356] "Generic (PLEG): container finished" podID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerID="851c9cc4d4a6f6fc8f991eed3a19789667eb405fdb6704abf32561ebf9e88c1e" exitCode=60
Nov 02 13:00:02.942773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.942695    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerDied","Data":"851c9cc4d4a6f6fc8f991eed3a19789667eb405fdb6704abf32561ebf9e88c1e"}
Nov 02 13:00:02.943019 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.942987    4928 scope.go:122] "RemoveContainer" containerID="851c9cc4d4a6f6fc8f991eed3a19789667eb405fdb6704abf32561ebf9e88c1e"
Nov 02 13:00:02.944769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.944741    4928 generic.go:356] "Generic (PLEG): container finished" podID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerID="f41a4e0f75556ed2f764e74cf2b42c95f90e4c6b90551c005b6f55fc14c708b4" exitCode=60
Nov 02 13:00:02.944922 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.944906    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerDied","Data":"f41a4e0f75556ed2f764e74cf2b42c95f90e4c6b90551c005b6f55fc14c708b4"}
Nov 02 13:00:02.945190 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:02.945174    4928 scope.go:122] "RemoveContainer" containerID="f41a4e0f75556ed2f764e74cf2b42c95f90e4c6b90551c005b6f55fc14c708b4"
Nov 02 13:00:03.021795 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.021742    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" podStartSLOduration=3.021724855 podStartE2EDuration="3.021724855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:03.021095251 +0000 UTC m=+2000.670149212" watchObservedRunningTime="2026-11-02 13:00:03.021724855 +0000 UTC m=+2000.670778815"
Nov 02 13:00:03.958137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.958030    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerStarted","Data":"60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"}
Nov 02 13:00:03.960806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.960773    4928 generic.go:356] "Generic (PLEG): container finished" podID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerID="4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d" exitCode=60
Nov 02 13:00:03.960978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.960924    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerDied","Data":"4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"}
Nov 02 13:00:03.960978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.960974    4928 scope.go:122] "RemoveContainer" containerID="c0593fe2f931ae16743a4d715459acd64a52b3729e1aaaca26a52a0133953646"
Nov 02 13:00:03.961300 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.961278    4928 scope.go:122] "RemoveContainer" containerID="4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"
Nov 02 13:00:03.964102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.963874    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerStarted","Data":"ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"}
Nov 02 13:00:03.966667 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:03.966410    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t_odh-ai-gateway-infra(7b68dec3-73cb-4d8a-aaf5-d929a03726db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db"
Nov 02 13:00:03.966819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:03.966793    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerStarted","Data":"692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"}
Nov 02 13:00:04.411165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.411117    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 13:00:04.411388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.411340    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy" containerID="cri-o://7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329" gracePeriod=30
Nov 02 13:00:04.980858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.980784    4928 scope.go:122] "RemoveContainer" containerID="4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"
Nov 02 13:00:04.981319 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:04.981260    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t_odh-ai-gateway-infra(7b68dec3-73cb-4d8a-aaf5-d929a03726db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db"
Nov 02 13:00:04.982468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.982444    4928 generic.go:356] "Generic (PLEG): container finished" podID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerID="ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10" exitCode=60
Nov 02 13:00:04.982595 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.982515    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerDied","Data":"ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"}
Nov 02 13:00:04.982595 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.982561    4928 scope.go:122] "RemoveContainer" containerID="851c9cc4d4a6f6fc8f991eed3a19789667eb405fdb6704abf32561ebf9e88c1e"
Nov 02 13:00:04.982820 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.982800    4928 scope.go:122] "RemoveContainer" containerID="ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"
Nov 02 13:00:04.983183 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:04.983161    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj_odh-ai-gateway-infra(153356b1-810d-4a44-ad73-e28bbd69fdec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec"
Nov 02 13:00:04.984776 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.984754    4928 generic.go:356] "Generic (PLEG): container finished" podID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerID="692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14" exitCode=60
Nov 02 13:00:04.984897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.984869    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerDied","Data":"692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"}
Nov 02 13:00:04.985142 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.985125    4928 scope.go:122] "RemoveContainer" containerID="692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"
Nov 02 13:00:04.985392 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:04.985372    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj_odh-ai-gateway-infra(c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f"
Nov 02 13:00:04.987005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.986981    4928 generic.go:356] "Generic (PLEG): container finished" podID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerID="60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1" exitCode=60
Nov 02 13:00:04.987091 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.987040    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerDied","Data":"60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"}
Nov 02 13:00:04.987276 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.987263    4928 scope.go:122] "RemoveContainer" containerID="60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"
Nov 02 13:00:04.987504 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:04.987486    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t_odh-ai-gateway-infra(06c8245e-f6b7-4aed-92d4-50bb39450819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819"
Nov 02 13:00:04.997307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:04.997287    4928 scope.go:122] "RemoveContainer" containerID="f41a4e0f75556ed2f764e74cf2b42c95f90e4c6b90551c005b6f55fc14c708b4"
Nov 02 13:00:05.012024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:05.012002    4928 scope.go:122] "RemoveContainer" containerID="3102d496a2e2060750d2b9f723736993967c4b50c1e1c1961c3ed9df818a6150"
Nov 02 13:00:05.997015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:05.996983    4928 scope.go:122] "RemoveContainer" containerID="60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"
Nov 02 13:00:05.997483 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:05.997346    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t_odh-ai-gateway-infra(06c8245e-f6b7-4aed-92d4-50bb39450819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819"
Nov 02 13:00:05.998794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:05.998768    4928 scope.go:122] "RemoveContainer" containerID="ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"
Nov 02 13:00:05.999086 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:05.999060    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj_odh-ai-gateway-infra(153356b1-810d-4a44-ad73-e28bbd69fdec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec"
Nov 02 13:00:06.000394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:06.000376    4928 scope.go:122] "RemoveContainer" containerID="692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"
Nov 02 13:00:06.000584 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:06.000565    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj_odh-ai-gateway-infra(c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f"
Nov 02 13:00:09.730759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.730713    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 13:00:09.746470 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:09.746432    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:00:09.850927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.850888    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.850938    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.850963    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.850984    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851032    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851054    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851153    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851238    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:09.851435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851350    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851383    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\") pod \"f9f3562b-200c-4e40-96ac-bc01741130b6\" (UID: \"f9f3562b-200c-4e40-96ac-bc01741130b6\") "
Nov 02 13:00:09.851435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851426    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:09.851702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851438    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:09.851702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851514    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:09.851840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.851694    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:09.852038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.852016    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3562b-200c-4e40-96ac-bc01741130b6-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.852132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.852043    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.852132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.852059    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.852132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.852071    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.852132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.852083    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.853743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.853717    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:09.853875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.853809    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:09.854078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.854049    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:00:09.854078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.854066    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9" pod "f9f3562b-200c-4e40-96ac-bc01741130b6" (UID: "f9f3562b-200c-4e40-96ac-bc01741130b6"). InnerVolumeSpecName "kube-api-access-j62b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:09.954417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.954322    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.954417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.954364    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j62b9\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-kube-api-access-j62b9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.954417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.954380    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.954417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.954393    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3562b-200c-4e40-96ac-bc01741130b6-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:09.992793 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:09.992760    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 13:00:10.041206 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.041169    4928 generic.go:356] "Generic (PLEG): container finished" podID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerID="7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329" exitCode=0
Nov 02 13:00:10.041421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.041401    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" event={"ID":"f9f3562b-200c-4e40-96ac-bc01741130b6","Type":"ContainerDied","Data":"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"}
Nov 02 13:00:10.041510 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.041475    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7" event={"ID":"f9f3562b-200c-4e40-96ac-bc01741130b6","Type":"ContainerDied","Data":"14b871caaca8a4906a5d9639c9c25b6b7d68727b18eaba295c5c76a8d0522e52"}
Nov 02 13:00:10.041510 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.041499    4928 scope.go:122] "RemoveContainer" containerID="7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"
Nov 02 13:00:10.041661 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.041649    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"
Nov 02 13:00:10.055168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.055144    4928 scope.go:122] "RemoveContainer" containerID="7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"
Nov 02 13:00:10.055539 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:10.055517    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329\": container with ID starting with 7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329 not found: ID does not exist" containerID="7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"
Nov 02 13:00:10.055624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.055546    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329"} err="failed to get container status \"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329\": rpc error: code = NotFound desc = could not find container \"7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329\": container with ID starting with 7603ea7726c7bfb9e02f3b58d5899fbea93921c92e7c1ba846e8987d6641c329 not found: ID does not exist"
Nov 02 13:00:10.101936 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.101900    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 13:00:10.104847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.104808    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-wjlc7"]
Nov 02 13:00:10.526689 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:10.526653    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" path="/var/lib/kubelet/pods/f9f3562b-200c-4e40-96ac-bc01741130b6/volumes"
Nov 02 13:00:14.490927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:14.490888    4928 scope.go:122] "RemoveContainer" containerID="ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"
Nov 02 13:00:14.491380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:14.491031    4928 scope.go:122] "RemoveContainer" containerID="4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"
Nov 02 13:00:15.111747 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:15.109720    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerStarted","Data":"9d9bbca8ab46173775b977061cb67c71fab837b9282e66791d97b5e74aab93f1"}
Nov 02 13:00:15.114995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:15.113182    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerStarted","Data":"ad9ab70b881e9f54d705a3738f162b97ce70436cf67c6b23693f91913f43ab10"}
Nov 02 13:00:16.122869 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.122787    4928 generic.go:356] "Generic (PLEG): container finished" podID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerID="9d9bbca8ab46173775b977061cb67c71fab837b9282e66791d97b5e74aab93f1" exitCode=60
Nov 02 13:00:16.123329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.122860    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerDied","Data":"9d9bbca8ab46173775b977061cb67c71fab837b9282e66791d97b5e74aab93f1"}
Nov 02 13:00:16.123329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.122929    4928 scope.go:122] "RemoveContainer" containerID="4d5992aa32439e0fe2fcad7bc01b7d985f4475816a4da5f0a46f9858c580171d"
Nov 02 13:00:16.123477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.123373    4928 scope.go:122] "RemoveContainer" containerID="9d9bbca8ab46173775b977061cb67c71fab837b9282e66791d97b5e74aab93f1"
Nov 02 13:00:16.123760 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:16.123731    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t_odh-ai-gateway-infra(7b68dec3-73cb-4d8a-aaf5-d929a03726db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db"
Nov 02 13:00:16.125328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.125304    4928 generic.go:356] "Generic (PLEG): container finished" podID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerID="ad9ab70b881e9f54d705a3738f162b97ce70436cf67c6b23693f91913f43ab10" exitCode=60
Nov 02 13:00:16.125430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.125367    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerDied","Data":"ad9ab70b881e9f54d705a3738f162b97ce70436cf67c6b23693f91913f43ab10"}
Nov 02 13:00:16.125856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.125805    4928 scope.go:122] "RemoveContainer" containerID="ad9ab70b881e9f54d705a3738f162b97ce70436cf67c6b23693f91913f43ab10"
Nov 02 13:00:16.126246 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:16.126218    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj_odh-ai-gateway-infra(153356b1-810d-4a44-ad73-e28bbd69fdec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec"
Nov 02 13:00:16.145268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.145239    4928 scope.go:122] "RemoveContainer" containerID="ebd200e0fd327233814264d3e4dd376b93459688247048d2d6798b42e188bf10"
Nov 02 13:00:16.167327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.167299    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:16.224316 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.224275    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:16.849698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.843345    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:00:16.849698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.844526    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy"
Nov 02 13:00:16.849698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.844550    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy"
Nov 02 13:00:16.849698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.844717    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f3562b-200c-4e40-96ac-bc01741130b6" containerName="istio-proxy"
Nov 02 13:00:16.854431 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.854400    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:16.882179 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.882142    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ecec44\""
Nov 02 13:00:16.885704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.884631    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ecec44-dockercfg-hx26q\""
Nov 02 13:00:16.885914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.885630    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:00:16.984547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.984486    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675dg\" (UniqueName: \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:16.985087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:16.985056    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.087011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.086974    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675dg\" (UniqueName: \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.087214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.087090    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.088015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.087990    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.112352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.112274    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-675dg\" (UniqueName: \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\") pod \"payload-processing-e2e-autores-ecec44-745988df69-7p8l8\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") " pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.201166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.201127    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:17.373412 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.373093    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:17.385538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.385421    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:17.492503 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.492474    4928 scope.go:122] "RemoveContainer" containerID="60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"
Nov 02 13:00:17.497032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.496955    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\") pod \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") "
Nov 02 13:00:17.497032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.497007    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\") pod \"153356b1-810d-4a44-ad73-e28bbd69fdec\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") "
Nov 02 13:00:17.497230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.497209    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\" (UniqueName: \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\") pod \"153356b1-810d-4a44-ad73-e28bbd69fdec\" (UID: \"153356b1-810d-4a44-ad73-e28bbd69fdec\") "
Nov 02 13:00:17.497378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.497359    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\" (UniqueName: \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\") pod \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\" (UID: \"7b68dec3-73cb-4d8a-aaf5-d929a03726db\") "
Nov 02 13:00:17.498619 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.498581    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca" pod "153356b1-810d-4a44-ad73-e28bbd69fdec" (UID: "153356b1-810d-4a44-ad73-e28bbd69fdec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:17.499156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.498958    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca" pod "7b68dec3-73cb-4d8a-aaf5-d929a03726db" (UID: "7b68dec3-73cb-4d8a-aaf5-d929a03726db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:17.501178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.501151    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k" pod "7b68dec3-73cb-4d8a-aaf5-d929a03726db" (UID: "7b68dec3-73cb-4d8a-aaf5-d929a03726db"). InnerVolumeSpecName "kube-api-access-92h4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:17.501651 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.501604    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8" pod "153356b1-810d-4a44-ad73-e28bbd69fdec" (UID: "153356b1-810d-4a44-ad73-e28bbd69fdec"). InnerVolumeSpecName "kube-api-access-48mm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:17.579158 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.579106    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:00:17.583587 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:17.583553    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ad1521_807a_432d_9179_2e2155ba8237.slice/crio-9195a4d304e00b53672241f3ab565370b6004bac854b414516a9778157213117 WatchSource:0}: Error finding container 9195a4d304e00b53672241f3ab565370b6004bac854b414516a9778157213117: Status 404 returned error can't find the container with id 9195a4d304e00b53672241f3ab565370b6004bac854b414516a9778157213117
Nov 02 13:00:17.599905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.599751    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mm8\" (UniqueName: \"kubernetes.io/projected/153356b1-810d-4a44-ad73-e28bbd69fdec-kube-api-access-48mm8\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:17.599905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.599781    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92h4k\" (UniqueName: \"kubernetes.io/projected/7b68dec3-73cb-4d8a-aaf5-d929a03726db-kube-api-access-92h4k\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:17.599905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.599796    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68dec3-73cb-4d8a-aaf5-d929a03726db-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:17.599905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:17.599844    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153356b1-810d-4a44-ad73-e28bbd69fdec-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.154814 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.154776    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj" event={"ID":"153356b1-810d-4a44-ad73-e28bbd69fdec","Type":"ContainerDied","Data":"e4f0c065307509cd2d30ff0a0bd8c8e5c823ea76560883bb85147f8da447ee7a"}
Nov 02 13:00:18.154996 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.154883    4928 scope.go:122] "RemoveContainer" containerID="ad9ab70b881e9f54d705a3738f162b97ce70436cf67c6b23693f91913f43ab10"
Nov 02 13:00:18.155044 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.155022    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"
Nov 02 13:00:18.159941 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.159842    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" event={"ID":"18ad1521-807a-432d-9179-2e2155ba8237","Type":"ContainerStarted","Data":"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"}
Nov 02 13:00:18.159941 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.159904    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" event={"ID":"18ad1521-807a-432d-9179-2e2155ba8237","Type":"ContainerStarted","Data":"9195a4d304e00b53672241f3ab565370b6004bac854b414516a9778157213117"}
Nov 02 13:00:18.162015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.161991    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:18.162740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.162714    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerStarted","Data":"86eaabc240f39ebedfec44c5763c61bdb2c0cea63925726814599afa4ade49bf"}
Nov 02 13:00:18.164220 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.164197    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t" event={"ID":"7b68dec3-73cb-4d8a-aaf5-d929a03726db","Type":"ContainerDied","Data":"f99db303376ef5ab4afac693cbfabd9512b9e63af81e31a34f2d67c55d4fe45d"}
Nov 02 13:00:18.164468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.164228    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"
Nov 02 13:00:18.172329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.172293    4928 scope.go:122] "RemoveContainer" containerID="9d9bbca8ab46173775b977061cb67c71fab837b9282e66791d97b5e74aab93f1"
Nov 02 13:00:18.228165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.228111    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" podStartSLOduration=2.228096995 podStartE2EDuration="2.228096995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:18.223408841 +0000 UTC m=+2015.872462800" watchObservedRunningTime="2026-11-02 13:00:18.228096995 +0000 UTC m=+2015.877150953"
Nov 02 13:00:18.282775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.282741    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:18.291702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.291668    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893740-wcnrj"]
Nov 02 13:00:18.385145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.384995    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:18.395070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.395034    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893740-stw8t"]
Nov 02 13:00:18.490066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.490034    4928 scope.go:122] "RemoveContainer" containerID="692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"
Nov 02 13:00:18.499006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.498979    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" path="/var/lib/kubelet/pods/153356b1-810d-4a44-ad73-e28bbd69fdec/volumes"
Nov 02 13:00:18.499423 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:18.499403    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" path="/var/lib/kubelet/pods/7b68dec3-73cb-4d8a-aaf5-d929a03726db/volumes"
Nov 02 13:00:19.180856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.180724    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerStarted","Data":"a0aede1e9cb37a9bb62e24fcf9a5f52836d821bd062ed458be2701dfd96d65e1"}
Nov 02 13:00:19.183038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.183005    4928 generic.go:356] "Generic (PLEG): container finished" podID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerID="86eaabc240f39ebedfec44c5763c61bdb2c0cea63925726814599afa4ade49bf" exitCode=60
Nov 02 13:00:19.183168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.183075    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerDied","Data":"86eaabc240f39ebedfec44c5763c61bdb2c0cea63925726814599afa4ade49bf"}
Nov 02 13:00:19.183168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.183125    4928 scope.go:122] "RemoveContainer" containerID="60f254a3cad3ed63a80f9ee8070197942d1e878303f89831c2f77d569e8e5ac1"
Nov 02 13:00:19.183503 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.183486    4928 scope.go:122] "RemoveContainer" containerID="86eaabc240f39ebedfec44c5763c61bdb2c0cea63925726814599afa4ade49bf"
Nov 02 13:00:19.183768 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:19.183748    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t_odh-ai-gateway-infra(06c8245e-f6b7-4aed-92d4-50bb39450819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819"
Nov 02 13:00:19.347678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:19.347639    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:20.193006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.192971    4928 generic.go:356] "Generic (PLEG): container finished" podID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerID="a0aede1e9cb37a9bb62e24fcf9a5f52836d821bd062ed458be2701dfd96d65e1" exitCode=60
Nov 02 13:00:20.193183 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.193047    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerDied","Data":"a0aede1e9cb37a9bb62e24fcf9a5f52836d821bd062ed458be2701dfd96d65e1"}
Nov 02 13:00:20.193183 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.193116    4928 scope.go:122] "RemoveContainer" containerID="692f40ff6606d91954fecd993f47996f0669009fbaa8be1722a09facd89a0a14"
Nov 02 13:00:20.194639 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.193541    4928 scope.go:122] "RemoveContainer" containerID="a0aede1e9cb37a9bb62e24fcf9a5f52836d821bd062ed458be2701dfd96d65e1"
Nov 02 13:00:20.194639 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:20.193939    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj_odh-ai-gateway-infra(c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f"
Nov 02 13:00:20.239256 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.239213    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:20.383211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.383178    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:20.472972 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.472869    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\") pod \"06c8245e-f6b7-4aed-92d4-50bb39450819\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") "
Nov 02 13:00:20.473147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.473106    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\" (UniqueName: \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\") pod \"06c8245e-f6b7-4aed-92d4-50bb39450819\" (UID: \"06c8245e-f6b7-4aed-92d4-50bb39450819\") "
Nov 02 13:00:20.473363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.473331    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca" pod "06c8245e-f6b7-4aed-92d4-50bb39450819" (UID: "06c8245e-f6b7-4aed-92d4-50bb39450819"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:20.473801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.473776    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c8245e-f6b7-4aed-92d4-50bb39450819-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.475734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.475707    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs" pod "06c8245e-f6b7-4aed-92d4-50bb39450819" (UID: "06c8245e-f6b7-4aed-92d4-50bb39450819"). InnerVolumeSpecName "kube-api-access-7dfbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:20.575161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:20.575119    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfbs\" (UniqueName: \"kubernetes.io/projected/06c8245e-f6b7-4aed-92d4-50bb39450819-kube-api-access-7dfbs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:21.209722 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.209692    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"
Nov 02 13:00:21.209911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.209732    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t" event={"ID":"06c8245e-f6b7-4aed-92d4-50bb39450819","Type":"ContainerDied","Data":"72d2b769dac3daad25df1e5b2c7a56ffca7aaee96521f8c97f442decdf8f758f"}
Nov 02 13:00:21.209911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.209802    4928 scope.go:122] "RemoveContainer" containerID="86eaabc240f39ebedfec44c5763c61bdb2c0cea63925726814599afa4ade49bf"
Nov 02 13:00:21.323474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.322013    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:21.326241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.325738    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e430c7-29893740-lsv9t"]
Nov 02 13:00:21.386520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.386491    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:21.493266 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.493242    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\" (UniqueName: \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\") pod \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") "
Nov 02 13:00:21.493440 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.493301    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\") pod \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\" (UID: \"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f\") "
Nov 02 13:00:21.493752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.493726    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca" pod "c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" (UID: "c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:21.495509 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.495486    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf" pod "c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" (UID: "c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f"). InnerVolumeSpecName "kube-api-access-k2jwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:21.594799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.594760    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jwf\" (UniqueName: \"kubernetes.io/projected/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-kube-api-access-k2jwf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:21.594799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:21.594790    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:22.247011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.246969    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj" event={"ID":"c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f","Type":"ContainerDied","Data":"8a530c64b0fc443421f971127f4c1cf2aed60400e1a2f74e267839b2f41c4334"}
Nov 02 13:00:22.247317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.247296    4928 scope.go:122] "RemoveContainer" containerID="a0aede1e9cb37a9bb62e24fcf9a5f52836d821bd062ed458be2701dfd96d65e1"
Nov 02 13:00:22.249255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.249230    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"
Nov 02 13:00:22.380185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.380118    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:22.386559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.386503    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893740-rd9pj"]
Nov 02 13:00:22.500047 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.499962    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" path="/var/lib/kubelet/pods/06c8245e-f6b7-4aed-92d4-50bb39450819/volumes"
Nov 02 13:00:22.500466 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:22.500451    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" path="/var/lib/kubelet/pods/c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f/volumes"
Nov 02 13:00:25.014818 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.014717    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" podUID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" containerName="main" containerID="cri-o://9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d" gracePeriod=45
Nov 02 13:00:25.328978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.328949    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 13:00:25.345246 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.345210    4928 generic.go:356] "Generic (PLEG): container finished" podID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" containerID="9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d" exitCode=0
Nov 02 13:00:25.345447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.345264    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"
Nov 02 13:00:25.345447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.345303    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" event={"ID":"6720dc5a-0e0a-44e0-b108-b0ef78768eea","Type":"ContainerDied","Data":"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"}
Nov 02 13:00:25.345447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.345395    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph" event={"ID":"6720dc5a-0e0a-44e0-b108-b0ef78768eea","Type":"ContainerDied","Data":"d93ee9ec7bbd941f7b8aac3b0e73562ce01058edff8f1ba98bd53c6b8fcbd452"}
Nov 02 13:00:25.345447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.345437    4928 scope.go:122] "RemoveContainer" containerID="9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"
Nov 02 13:00:25.358142 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.358125    4928 scope.go:122] "RemoveContainer" containerID="9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"
Nov 02 13:00:25.358456 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:25.358433    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d\": container with ID starting with 9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d not found: ID does not exist" containerID="9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"
Nov 02 13:00:25.358536 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.358463    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d"} err="failed to get container status \"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d\": rpc error: code = NotFound desc = could not find container \"9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d\": container with ID starting with 9828c8d6cd7cd1fac28900db0b830562f787cab663cf24398f31664c114abe7d not found: ID does not exist"
Nov 02 13:00:25.378892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.378865    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.378952    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\" (UniqueName: \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.378994    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379019    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\" (UniqueName: \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379109    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379138    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\") pod \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\" (UID: \"6720dc5a-0e0a-44e0-b108-b0ef78768eea\") "
Nov 02 13:00:25.379283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379194    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.379458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379430    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.379681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379627    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.379968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379912    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.379968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379934    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.379968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.379948    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.381867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.381586    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.381867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.381662    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "kube-api-access-xrckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:25.381867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.381679    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs" pod "6720dc5a-0e0a-44e0-b108-b0ef78768eea" (UID: "6720dc5a-0e0a-44e0-b108-b0ef78768eea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:25.481707 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.481663    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6720dc5a-0e0a-44e0-b108-b0ef78768eea-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.481707 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.481698    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6720dc5a-0e0a-44e0-b108-b0ef78768eea-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.481707 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.481712    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrckf\" (UniqueName: \"kubernetes.io/projected/6720dc5a-0e0a-44e0-b108-b0ef78768eea-kube-api-access-xrckf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.695114 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.695073    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 13:00:25.708259 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:25.708221    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6416f0/auth-test-model-6416f0-kserve-74f98db469-djqph"]
Nov 02 13:00:26.498678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:26.498642    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" path="/var/lib/kubelet/pods/6720dc5a-0e0a-44e0-b108-b0ef78768eea/volumes"
Nov 02 13:00:29.187282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:29.187251    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:00:31.580004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:31.579968    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 13:00:31.580479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:31.580212    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy" containerID="cri-o://36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501" gracePeriod=30
Nov 02 13:00:36.856002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.855927    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 13:00:36.963574 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963532    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.963790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963626    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.963790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963671    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.963790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963751    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.964041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963932    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.964041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.963991    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.964041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.964028    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.964210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.964142    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.964268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.964217    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\") pod \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\" (UID: \"dae0248a-61fb-4d8b-9d0f-44a21278cf50\") "
Nov 02 13:00:36.965352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.965049    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:36.965352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.965104    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:36.965803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.965757    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:36.965928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.965809    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:36.965928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.965856    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:36.968329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.967742    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:00:36.976283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.976251    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:36.976695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.976672    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:36.979987 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:36.979961    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79" pod "dae0248a-61fb-4d8b-9d0f-44a21278cf50" (UID: "dae0248a-61fb-4d8b-9d0f-44a21278cf50"). InnerVolumeSpecName "kube-api-access-mnb79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:37.066254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066220    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066251    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066260    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066270    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb79\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-kube-api-access-mnb79\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066279    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066288    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066296    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066303    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.066492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.066310    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0248a-61fb-4d8b-9d0f-44a21278cf50-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:00:37.441892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.441819    4928 generic.go:356] "Generic (PLEG): container finished" podID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerID="36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501" exitCode=0
Nov 02 13:00:37.442101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.441940    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" event={"ID":"dae0248a-61fb-4d8b-9d0f-44a21278cf50","Type":"ContainerDied","Data":"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"}
Nov 02 13:00:37.442101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.442005    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr" event={"ID":"dae0248a-61fb-4d8b-9d0f-44a21278cf50","Type":"ContainerDied","Data":"72764de59e94e96cf9cc0435f66a13cd4ea0c65b45010caf8e005c9c494847ea"}
Nov 02 13:00:37.442101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.442032    4928 scope.go:122] "RemoveContainer" containerID="36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"
Nov 02 13:00:37.442351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.442329    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"
Nov 02 13:00:37.458302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.458276    4928 scope.go:122] "RemoveContainer" containerID="36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"
Nov 02 13:00:37.460330 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:37.460279    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501\": container with ID starting with 36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501 not found: ID does not exist" containerID="36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"
Nov 02 13:00:37.460459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.460326    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501"} err="failed to get container status \"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501\": rpc error: code = NotFound desc = could not find container \"36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501\": container with ID starting with 36bfc8452ced41adbcb0be38acff1f8a6fdc8e389d7676249e60ee86532b9501 not found: ID does not exist"
Nov 02 13:00:37.514381 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.514347    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 13:00:37.528650 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:37.528617    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nqrlr"]
Nov 02 13:00:38.498162 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:38.498122    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" path="/var/lib/kubelet/pods/dae0248a-61fb-4d8b-9d0f-44a21278cf50/volumes"
Nov 02 13:00:39.729844 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:39.729794    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:00:40.708888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.708847    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:00:40.709522 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709500    4928 cpu_manager.go:414] "containerMap: removing container" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709523    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709573    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709582    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709608    4928 cpu_manager.go:414] "containerMap: removing container" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709615    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709628    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709635    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709645    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.709652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709653    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709668    4928 cpu_manager.go:414] "containerMap: removing container" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709675    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709695    4928 cpu_manager.go:414] "containerMap: removing container" podUID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" containerName="main"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709704    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" containerName="main"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709720    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709728    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709746    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709753    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709769    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709776    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709786    4928 cpu_manager.go:414] "containerMap: removing container" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709793    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709803    4928 cpu_manager.go:414] "containerMap: removing container" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709809    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709976    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.709991    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710002    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710010    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710021    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710030    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710043    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710060    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae0248a-61fb-4d8b-9d0f-44a21278cf50" containerName="istio-proxy"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710069    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710076    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:40.710124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.710086    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="6720dc5a-0e0a-44e0-b108-b0ef78768eea" containerName="main"
Nov 02 13:00:40.715421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.715401    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.733356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.732179    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ecec44-serving-cert\""
Nov 02 13:00:40.752154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.752117    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:00:40.810422 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.810376    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.810422 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.810422    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9q9\" (UniqueName: \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.810676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.810649    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.810778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.810762    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.912155    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.912217    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9q9\" (UniqueName: \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.912387    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.912472    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:40.912668    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-serving-cert: secret "maas-api-e2e-autores-ecec44-serving-cert" not found
Nov 02 13:00:40.913052 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:00:40.912747    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls podName:5202c1f9-f3a6-443e-ad9b-8254457cf1d9 nodeName:}" failed. No retries permitted until 2026-11-02 13:00:41.412727196 +0000 UTC m=+2039.061781150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls") pod "maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" (UID: "5202c1f9-f3a6-443e-ad9b-8254457cf1d9") : secret "maas-api-e2e-autores-ecec44-serving-cert" not found
Nov 02 13:00:40.913494 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.913177    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.916256 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.915792    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:40.940569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:40.940531    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9q9\" (UniqueName: \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:41.420269 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:41.420231    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:41.422849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:41.422811    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") pod \"maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:41.633024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:41.632987    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:41.949717 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:41.949683    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:00:41.950635 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:41.950607    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5202c1f9_f3a6_443e_ad9b_8254457cf1d9.slice/crio-4a3ad5a0f9eeac59c2b2e08d8c79a7bbc83fbecb0262ad57d0cb5b1b24577093 WatchSource:0}: Error finding container 4a3ad5a0f9eeac59c2b2e08d8c79a7bbc83fbecb0262ad57d0cb5b1b24577093: Status 404 returned error can't find the container with id 4a3ad5a0f9eeac59c2b2e08d8c79a7bbc83fbecb0262ad57d0cb5b1b24577093
Nov 02 13:00:41.952134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:41.952111    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:00:42.514802 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:42.514755    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" event={"ID":"5202c1f9-f3a6-443e-ad9b-8254457cf1d9","Type":"ContainerStarted","Data":"4a3ad5a0f9eeac59c2b2e08d8c79a7bbc83fbecb0262ad57d0cb5b1b24577093"}
Nov 02 13:00:43.520674 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.520629    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" event={"ID":"5202c1f9-f3a6-443e-ad9b-8254457cf1d9","Type":"ContainerStarted","Data":"c3f2313d00343321ea4287f1c2e0649b0312abbf6194e6f8b8c0154863473729"}
Nov 02 13:00:43.521311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.520732    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:00:43.530406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.530363    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:00:43.531055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531036    4928 cpu_manager.go:414] "containerMap: removing container" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:43.531055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531055    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:43.531257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531070    4928 cpu_manager.go:414] "containerMap: removing container" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:43.531257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531076    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c8245e-f6b7-4aed-92d4-50bb39450819" containerName="cleanup"
Nov 02 13:00:43.531257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531231    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61d16c0-f80c-4f7a-89e0-4d7a8ca4f06f" containerName="cleanup"
Nov 02 13:00:43.531257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531245    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68dec3-73cb-4d8a-aaf5-d929a03726db" containerName="cleanup"
Nov 02 13:00:43.531257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.531256    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="153356b1-810d-4a44-ad73-e28bbd69fdec" containerName="cleanup"
Nov 02 13:00:43.535111 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.535086    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.551528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.551504    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25f7a6-openshift-default-dockercfg-z957v\""
Nov 02 13:00:43.560233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.560198    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:00:43.571562 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.571485    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" podStartSLOduration=3.00177996 podStartE2EDuration="3.571466841s" totalImagesPullingTime="569.686881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:43.570577674 +0000 UTC m=+2041.219631629" watchObservedRunningTime="2026-11-02 13:00:43.571466841 +0000 UTC m=+2041.220520801"
Nov 02 13:00:43.650643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.650592    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.650866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.650657    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.650924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.650898    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.650962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.650936    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.650998 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.650965    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.651057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.651038    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.651097 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.651076    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.651133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.651103    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.651198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.651184    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bht\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753557    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753637    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753667    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753717    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753755    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.753787    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.754241    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bht\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.754321    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.754383    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.755279    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.763136    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.763796    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.764116    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.766851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.765369    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.781381 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.781288    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.781591 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.781559    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.801761 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.801727    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.806929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.806847    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bht\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.857569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.857529    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:43.886935 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:43.886907    4928 scope.go:122] "RemoveContainer" containerID="a143890251d8ff18dd841e913d3f5ebe21d5cf0b1d4dbaeb4939cd0324dbe818"
Nov 02 13:00:44.191796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.191766    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:00:44.192400 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:00:44.192367    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107a0384_09e7_42d3_902e_3f506d80c2fb.slice/crio-6db7f652ed38544ff805ada6f648a00395daf1b67d59f48b0b5c73b76e313b17 WatchSource:0}: Error finding container 6db7f652ed38544ff805ada6f648a00395daf1b67d59f48b0b5c73b76e313b17: Status 404 returned error can't find the container with id 6db7f652ed38544ff805ada6f648a00395daf1b67d59f48b0b5c73b76e313b17
Nov 02 13:00:44.195101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.195068    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:00:44.195178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.195129    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:00:44.195178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.195158    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:00:44.530344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.530294    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerStarted","Data":"9a9a831118192fd865132e9f83f48e3fef88eda98daa2f5f41b269cc3e2e647c"}
Nov 02 13:00:44.530894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.530359    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerStarted","Data":"6db7f652ed38544ff805ada6f648a00395daf1b67d59f48b0b5c73b76e313b17"}
Nov 02 13:00:44.581977 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.581911    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podStartSLOduration=1.581892278 podStartE2EDuration="1.581892278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:44.578177153 +0000 UTC m=+2042.227231105" watchObservedRunningTime="2026-11-02 13:00:44.581892278 +0000 UTC m=+2042.230946238"
Nov 02 13:00:44.857952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:44.857867    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:45.865572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:45.865530    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:46.550339 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:46.550310    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:46.551411 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:46.551386    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:00:49.546487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:00:49.546457    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:01:09.804812 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:01:09.804767    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:01:33.752878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.752836    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:01:33.758579 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.758554    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.794744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.794712    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:01:33.803438 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.803395    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-00d004-openshift-default-dockercfg-nwdfh\""
Nov 02 13:01:33.925326 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925285    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.925695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925665    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.925845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925796    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.925930 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925852    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.925930 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925879    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.925930 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.925913    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.926126 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.926045    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.926126 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.926075    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:33.926126 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:33.926103    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s8d\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.027904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.027778    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.027904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.027821    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.027904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.027885    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.027978    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.028001    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.028019    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s8d\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.028048    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.028572    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.028863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.028816    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.043921 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.031624    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.043921 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.032904    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.043921 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.036594    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.044917 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.044864    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.045228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.045072    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.055231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.055196    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.064878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.058705    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.086849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.071423    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.097167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.094591    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s8d\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\") pod \"e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") " pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.375289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.375202    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:34.710172 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.710141    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:01:34.711999 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:01:34.711969    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474cc566_a72a_4bf1_b668_eb05ae7985a8.slice/crio-7797a5dc31f5d57dc99208380dbc01e05689a8fbd4c1be7c614d291898c499eb WatchSource:0}: Error finding container 7797a5dc31f5d57dc99208380dbc01e05689a8fbd4c1be7c614d291898c499eb: Status 404 returned error can't find the container with id 7797a5dc31f5d57dc99208380dbc01e05689a8fbd4c1be7c614d291898c499eb
Nov 02 13:01:34.715237 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.715196    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:34.715351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.715301    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:34.715351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.715345    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:34.952850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.951864    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerStarted","Data":"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"}
Nov 02 13:01:34.952850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:34.951930    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerStarted","Data":"7797a5dc31f5d57dc99208380dbc01e05689a8fbd4c1be7c614d291898c499eb"}
Nov 02 13:01:35.005728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:35.005660    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podStartSLOduration=2.0056461 podStartE2EDuration="2.0056461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:35.000933997 +0000 UTC m=+2092.649987977" watchObservedRunningTime="2026-11-02 13:01:35.0056461 +0000 UTC m=+2092.654700057"
Nov 02 13:01:35.376328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:35.376244    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:35.382624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:35.382598    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:35.966000 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:35.961908    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:35.967070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:35.967043    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:01:39.866311 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:01:39.866272    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:01:42.967895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:42.967867    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:01:42.970072 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:42.970049    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:01:47.592995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:47.592952    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:01:47.593395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:47.593032    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:01:49.139318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.139279    4928 generic.go:356] "Generic (PLEG): container finished" podID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerID="9a9a831118192fd865132e9f83f48e3fef88eda98daa2f5f41b269cc3e2e647c" exitCode=0
Nov 02 13:01:49.140235 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.139353    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerDied","Data":"9a9a831118192fd865132e9f83f48e3fef88eda98daa2f5f41b269cc3e2e647c"}
Nov 02 13:01:49.140235 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.139835    4928 scope.go:122] "RemoveContainer" containerID="9a9a831118192fd865132e9f83f48e3fef88eda98daa2f5f41b269cc3e2e647c"
Nov 02 13:01:49.141947 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.141910    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:49.142062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.141987    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:49.142062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.142021    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:50.006850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.993616    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:01:50.006850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:49.999222    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.052847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.046191    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e422b-openshift-default-dockercfg-gh8mv\""
Nov 02 13:01:50.075886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.072231    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:01:50.093934 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.093876    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.093985    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094089    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094288 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094240    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094294    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094452    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nllw\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094551    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094730    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.094868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.094809    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.189071 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.189035    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerStarted","Data":"015e75f7718de83aab5004221260152d3f81c18d4bb0d9531dcee0339aa16706"}
Nov 02 13:01:50.196071 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196031    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196087    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196116    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196164    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196198    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196266    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nllw\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196300    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196398    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.196493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.196437    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.199260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.199232    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.199451 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.199287    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.199567 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.199394    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.199798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.199768    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.199911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.199869    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.200164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.200137    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.200697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.200672    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.223580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.223542    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.223768 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.223660    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nllw\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\") pod \"e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") " pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.381175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.381082    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:50.764286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.764250    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:01:50.764692 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:01:50.764665    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d55fcd_fa53_452a_bb7e_1134b9581b6a.slice/crio-a5e98587ad931af053707795cb656f51467c14f8da2cce4e0d90535181c9fc9e WatchSource:0}: Error finding container a5e98587ad931af053707795cb656f51467c14f8da2cce4e0d90535181c9fc9e: Status 404 returned error can't find the container with id a5e98587ad931af053707795cb656f51467c14f8da2cce4e0d90535181c9fc9e
Nov 02 13:01:50.768031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.767989    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:50.768137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.768078    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:50.768137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.768121    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:01:50.859931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.858297    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:01:50.859931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.858623    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 02 13:01:50.859931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:50.858677    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 02 13:01:51.199676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.199586    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerStarted","Data":"d618487c66deff3093ffb75e7c9932be319202e9e0ae11309ddda238e898aa49"}
Nov 02 13:01:51.200255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.200228    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerStarted","Data":"a5e98587ad931af053707795cb656f51467c14f8da2cce4e0d90535181c9fc9e"}
Nov 02 13:01:51.382861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.382115    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:51.389166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.388918    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:51.447212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.447142    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podStartSLOduration=2.447120233 podStartE2EDuration="2.447120233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:51.262329378 +0000 UTC m=+2108.911383338" watchObservedRunningTime="2026-11-02 13:01:51.447120233 +0000 UTC m=+2109.096174193"
Nov 02 13:01:51.858957 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.858916    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 02 13:01:51.859163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:51.858985    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 02 13:01:52.209061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:52.208952    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:52.209951 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:52.209932    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:01:52.859165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:52.859125    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 02 13:01:52.859379 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:52.859205    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 02 13:01:53.858597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:53.858503    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 02 13:01:53.858597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:53.858579    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 02 13:01:54.858158 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:54.858119    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 02 13:01:54.858339 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:54.858183    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 02 13:01:55.936219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:55.932859    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:01:55.949310 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:55.941689    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:01:55.955353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:01:55.955171    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:02:03.911928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:03.911892    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:02:04.824798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.824762    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:02:04.830179 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.830155    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.847686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.847664    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ef5efe-openshift-default-dockercfg-4cfq2\""
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.887970    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888061    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888123    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888242    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888363    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4lx\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888470    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888502    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888604    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888673    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:04.891847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:04.888704    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006559    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006643    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006710    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006776    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4lx\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006886    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.011853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.006914    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.062849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.060591    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.074322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.074282    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.093418 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.085630    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.107659 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.107616    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.127889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.125717    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.127889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.125910    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.127889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.125945    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.127889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.126406    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.129556 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.129509    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.130144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.130062    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.137744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.137657    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4lx\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.166242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.159321    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.446233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.446129    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:05.781195 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.781156    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:02:05.787140 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:02:05.787108    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7381c74f_d296_49a1_9f74_66f1838f7c1b.slice/crio-008b68326efa6e0003bac559c86d18c5e4df816d81aae380ee0cc9fb5b2c4c4d WatchSource:0}: Error finding container 008b68326efa6e0003bac559c86d18c5e4df816d81aae380ee0cc9fb5b2c4c4d: Status 404 returned error can't find the container with id 008b68326efa6e0003bac559c86d18c5e4df816d81aae380ee0cc9fb5b2c4c4d
Nov 02 13:02:05.790635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.790598    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:05.790740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.790671    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:05.790740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:05.790707    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:06.369903 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:06.369845    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerStarted","Data":"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"}
Nov 02 13:02:06.370374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:06.369920    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerStarted","Data":"008b68326efa6e0003bac559c86d18c5e4df816d81aae380ee0cc9fb5b2c4c4d"}
Nov 02 13:02:06.435015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:06.434939    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podStartSLOduration=2.434902367 podStartE2EDuration="2.434902367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:06.431894099 +0000 UTC m=+2124.080948061" watchObservedRunningTime="2026-11-02 13:02:06.434902367 +0000 UTC m=+2124.083956325"
Nov 02 13:02:06.446683 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:06.446639    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:06.454438 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:06.454399    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:07.377893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:07.377858    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:07.378866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:07.378815    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:02:08.912904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:08.912856    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" podUID="18ad1521-807a-432d-9179-2e2155ba8237" containerName="payload-processing" containerID="cri-o://1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41" gracePeriod=25
Nov 02 13:02:09.215878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.215850    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:02:09.315552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.314246    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\") pod \"18ad1521-807a-432d-9179-2e2155ba8237\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") "
Nov 02 13:02:09.315552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.314451    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\" (UniqueName: \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\") pod \"18ad1521-807a-432d-9179-2e2155ba8237\" (UID: \"18ad1521-807a-432d-9179-2e2155ba8237\") "
Nov 02 13:02:09.317397 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.317363    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume" pod "18ad1521-807a-432d-9179-2e2155ba8237" (UID: "18ad1521-807a-432d-9179-2e2155ba8237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:09.318210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.318179    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg" pod "18ad1521-807a-432d-9179-2e2155ba8237" (UID: "18ad1521-807a-432d-9179-2e2155ba8237"). InnerVolumeSpecName "kube-api-access-675dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:09.394464 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.394431    4928 generic.go:356] "Generic (PLEG): container finished" podID="18ad1521-807a-432d-9179-2e2155ba8237" containerID="1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41" exitCode=0
Nov 02 13:02:09.394623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.394475    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"
Nov 02 13:02:09.394623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.394518    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" event={"ID":"18ad1521-807a-432d-9179-2e2155ba8237","Type":"ContainerDied","Data":"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"}
Nov 02 13:02:09.394623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.394576    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" event={"ID":"18ad1521-807a-432d-9179-2e2155ba8237","Type":"ContainerDied","Data":"9195a4d304e00b53672241f3ab565370b6004bac854b414516a9778157213117"}
Nov 02 13:02:09.394623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.394599    4928 scope.go:122] "RemoveContainer" containerID="1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"
Nov 02 13:02:09.405555 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.405531    4928 scope.go:122] "RemoveContainer" containerID="1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"
Nov 02 13:02:09.405878 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:02:09.405857    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41\": container with ID starting with 1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41 not found: ID does not exist" containerID="1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"
Nov 02 13:02:09.405967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.405890    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41"} err="failed to get container status \"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41\": rpc error: code = NotFound desc = could not find container \"1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41\": container with ID starting with 1cd6d9d5fee7b2cff785433b58cb18438ff39f61d01acbe7df6c526d5b768d41 not found: ID does not exist"
Nov 02 13:02:09.416238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.416216    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ad1521-807a-432d-9179-2e2155ba8237-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:09.416238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.416240    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-675dg\" (UniqueName: \"kubernetes.io/projected/18ad1521-807a-432d-9179-2e2155ba8237-kube-api-access-675dg\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:09.452400 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.452306    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:02:09.455996 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.455963    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8"]
Nov 02 13:02:09.508086 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.508055    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:02:09.508679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.508666    4928 cpu_manager.go:414] "containerMap: removing container" podUID="18ad1521-807a-432d-9179-2e2155ba8237" containerName="payload-processing"
Nov 02 13:02:09.508727 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.508681    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ad1521-807a-432d-9179-2e2155ba8237" containerName="payload-processing"
Nov 02 13:02:09.508806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.508796    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ad1521-807a-432d-9179-2e2155ba8237" containerName="payload-processing"
Nov 02 13:02:09.513602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.513583    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.533186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.531722    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-00d004-serving-cert\""
Nov 02 13:02:09.542262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.542236    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:02:09.618729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.618692    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2w4\" (UniqueName: \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.618925 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.618818    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.619032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.618936    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.619032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.619009    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.723037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.722938    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.724034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.723996    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2w4\" (UniqueName: \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.724228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.724130    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.724228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.724180    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.730937 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.729743    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.733459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.733429    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.735469 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.735439    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.754890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.754425    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2w4\" (UniqueName: \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\") pod \"maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:09.767184 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:02:09.767146    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:02:09.838989 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:09.838950    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:10.174735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:10.174705    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:02:10.175209 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:02:10.175186    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067983aa_fbee_4750_83ae_3bbbda43fc0a.slice/crio-a80fb15939f333ae457e60a3801ca2afde14d7158f4b79a61e82f2ad34ff542d WatchSource:0}: Error finding container a80fb15939f333ae457e60a3801ca2afde14d7158f4b79a61e82f2ad34ff542d: Status 404 returned error can't find the container with id a80fb15939f333ae457e60a3801ca2afde14d7158f4b79a61e82f2ad34ff542d
Nov 02 13:02:10.184000 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:10.183963    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-ecec44-745988df69-7p8l8" podUID="18ad1521-807a-432d-9179-2e2155ba8237" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 13:02:10.184128 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:02:10.184087    4928 logging.go:55] [core] [Channel #3481 SubChannel #3482]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.172:50052", ServerName: "10.132.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.172:50052: operation was canceled"
Nov 02 13:02:10.405104 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:10.405069    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" event={"ID":"067983aa-fbee-4750-83ae-3bbbda43fc0a","Type":"ContainerStarted","Data":"a80fb15939f333ae457e60a3801ca2afde14d7158f4b79a61e82f2ad34ff542d"}
Nov 02 13:02:10.500174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:10.500130    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ad1521-807a-432d-9179-2e2155ba8237" path="/var/lib/kubelet/pods/18ad1521-807a-432d-9179-2e2155ba8237/volumes"
Nov 02 13:02:11.413952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.413915    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" event={"ID":"067983aa-fbee-4750-83ae-3bbbda43fc0a","Type":"ContainerStarted","Data":"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"}
Nov 02 13:02:11.414383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.414041    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:11.460989 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.460936    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" podStartSLOduration=2.073814561 podStartE2EDuration="2.460921978s" totalImagesPullingTime="387.107417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:11.459327959 +0000 UTC m=+2129.108381922" watchObservedRunningTime="2026-11-02 13:02:11.460921978 +0000 UTC m=+2129.109975935"
Nov 02 13:02:11.808032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.807994    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:11.812671 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.812645    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.838759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.835691    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ecec44-openshift-default-dockercfg-s57mf\""
Nov 02 13:02:11.850984 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.850941    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:11.875171 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875127    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875191    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875298    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875499 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875391    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875561    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wlk\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875636    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875724    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.875951 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875773    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.876016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.875999    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977175    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977270    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977292    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977447    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977603    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wlk\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977671    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977685    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977703    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977743    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.977886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.977759    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.978191 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.978159    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.978334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.978310    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.978413 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.978329    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.980630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.980601    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:11.982204 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:11.982158    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:12.006642 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.006606    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:12.006806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.006729    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wlk\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:12.128735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.128630    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:12.522013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.521973    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:12.550848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.536656    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:12.550848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.536839    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:12.550848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:12.536914    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:13.286647 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.286612    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:02:13.286900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.286861    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" podUID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" containerName="maas-api" containerID="cri-o://c3f2313d00343321ea4287f1c2e0649b0312abbf6194e6f8b8c0154863473729" gracePeriod=30
Nov 02 13:02:13.462847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.462185    4928 generic.go:356] "Generic (PLEG): container finished" podID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" containerID="c3f2313d00343321ea4287f1c2e0649b0312abbf6194e6f8b8c0154863473729" exitCode=0
Nov 02 13:02:13.462847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.462293    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" event={"ID":"5202c1f9-f3a6-443e-ad9b-8254457cf1d9","Type":"ContainerDied","Data":"c3f2313d00343321ea4287f1c2e0649b0312abbf6194e6f8b8c0154863473729"}
Nov 02 13:02:13.469593 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.469546    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" event={"ID":"b00b6cfc-4c1f-4911-b046-0438982d2379","Type":"ContainerStarted","Data":"350b8421f5644782468c499256f734b31c8de1428a0baa38fc3767f67d4eb22c"}
Nov 02 13:02:13.469775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.469622    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" event={"ID":"b00b6cfc-4c1f-4911-b046-0438982d2379","Type":"ContainerStarted","Data":"2d580a7e0d6fb5fb2e406733b67881b1f8d12a1eaa5a57f575bbee831062e94b"}
Nov 02 13:02:13.563909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.561976    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podStartSLOduration=2.561956312 podStartE2EDuration="2.561956312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:13.561677289 +0000 UTC m=+2131.210731247" watchObservedRunningTime="2026-11-02 13:02:13.561956312 +0000 UTC m=+2131.211010271"
Nov 02 13:02:13.913370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:13.913339    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:02:14.064693 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.064657    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\" (UniqueName: \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\") pod \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") "
Nov 02 13:02:14.064693 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.064697    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") pod \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") "
Nov 02 13:02:14.064984 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.064845    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\") pod \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") "
Nov 02 13:02:14.064984 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.064887    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\") pod \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\" (UID: \"5202c1f9-f3a6-443e-ad9b-8254457cf1d9\") "
Nov 02 13:02:14.065328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.065297    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca" pod "5202c1f9-f3a6-443e-ad9b-8254457cf1d9" (UID: "5202c1f9-f3a6-443e-ad9b-8254457cf1d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:14.065632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.065611    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:14.067101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.067070    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls" pod "5202c1f9-f3a6-443e-ad9b-8254457cf1d9" (UID: "5202c1f9-f3a6-443e-ad9b-8254457cf1d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:02:14.067515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.067494    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9" pod "5202c1f9-f3a6-443e-ad9b-8254457cf1d9" (UID: "5202c1f9-f3a6-443e-ad9b-8254457cf1d9"). InnerVolumeSpecName "kube-api-access-zg9q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:14.067577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.067559    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs" pod "5202c1f9-f3a6-443e-ad9b-8254457cf1d9" (UID: "5202c1f9-f3a6-443e-ad9b-8254457cf1d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:02:14.137904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.128989    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:14.137904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.131322    4928 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 13:02:14.137904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.131377    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 13:02:14.167750 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.167670    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:14.167750 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.167708    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9q9\" (UniqueName: \"kubernetes.io/projected/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-kube-api-access-zg9q9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:14.167750 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.167723    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5202c1f9-f3a6-443e-ad9b-8254457cf1d9-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:14.479343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.479256    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"
Nov 02 13:02:14.479343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.479291    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69" event={"ID":"5202c1f9-f3a6-443e-ad9b-8254457cf1d9","Type":"ContainerDied","Data":"4a3ad5a0f9eeac59c2b2e08d8c79a7bbc83fbecb0262ad57d0cb5b1b24577093"}
Nov 02 13:02:14.479584 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.479353    4928 scope.go:122] "RemoveContainer" containerID="c3f2313d00343321ea4287f1c2e0649b0312abbf6194e6f8b8c0154863473729"
Nov 02 13:02:14.561322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.561280    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:02:14.567292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:14.567260    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ecec44-f4bbcb6b6-zgp69"]
Nov 02 13:02:15.134852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:15.129192    4928 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 13:02:15.134852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:15.129286    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 13:02:16.142851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:16.129908    4928 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 13:02:16.142851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:16.129976    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 13:02:16.500400 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:16.500366    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" path="/var/lib/kubelet/pods/5202c1f9-f3a6-443e-ad9b-8254457cf1d9/volumes"
Nov 02 13:02:17.137852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:17.131253    4928 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 13:02:17.137852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:17.131327    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 13:02:17.450904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:17.448324    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:02:18.141850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:18.129185    4928 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 13:02:18.141850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:18.129245    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 13:02:18.799628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:18.799588    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:18.800163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:18.799797    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy" containerID="cri-o://350b8421f5644782468c499256f734b31c8de1428a0baa38fc3767f67d4eb22c" gracePeriod=30
Nov 02 13:02:24.573976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.573938    4928 generic.go:356] "Generic (PLEG): container finished" podID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerID="350b8421f5644782468c499256f734b31c8de1428a0baa38fc3767f67d4eb22c" exitCode=0
Nov 02 13:02:24.574496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.574005    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" event={"ID":"b00b6cfc-4c1f-4911-b046-0438982d2379","Type":"ContainerDied","Data":"350b8421f5644782468c499256f734b31c8de1428a0baa38fc3767f67d4eb22c"}
Nov 02 13:02:24.692678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.692646    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:24.795888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.795814    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796056    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.796333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796212    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796328    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796460 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796391    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796460 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796453    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796511    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796541    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796701 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796593    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796701 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796680    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\") pod \"b00b6cfc-4c1f-4911-b046-0438982d2379\" (UID: \"b00b6cfc-4c1f-4911-b046-0438982d2379\") "
Nov 02 13:02:24.796847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.796804    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.797095 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797071    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.797307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797288    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.797413 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797393    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.797474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797419    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.797474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797433    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.797474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.797410    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:24.799692 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.799658    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.800979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.800958    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:02:24.801069 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.801011    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:24.801069 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.801051    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk" pod "b00b6cfc-4c1f-4911-b046-0438982d2379" (UID: "b00b6cfc-4c1f-4911-b046-0438982d2379"). InnerVolumeSpecName "kube-api-access-s2wlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:24.899182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899132    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.899182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899175    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.899182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899191    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wlk\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-kube-api-access-s2wlk\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.899429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899203    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00b6cfc-4c1f-4911-b046-0438982d2379-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.899429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899215    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.899429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:24.899227    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00b6cfc-4c1f-4911-b046-0438982d2379-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:02:25.583708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.583673    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"
Nov 02 13:02:25.584200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.583675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv" event={"ID":"b00b6cfc-4c1f-4911-b046-0438982d2379","Type":"ContainerDied","Data":"2d580a7e0d6fb5fb2e406733b67881b1f8d12a1eaa5a57f575bbee831062e94b"}
Nov 02 13:02:25.584200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.583855    4928 scope.go:122] "RemoveContainer" containerID="350b8421f5644782468c499256f734b31c8de1428a0baa38fc3767f67d4eb22c"
Nov 02 13:02:25.655299 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.655245    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:25.668939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.668901    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-bw9fv"]
Nov 02 13:02:25.927863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.927738    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:02:25.928406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928390    4928 cpu_manager.go:414] "containerMap: removing container" podUID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" containerName="maas-api"
Nov 02 13:02:25.928461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928408    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" containerName="maas-api"
Nov 02 13:02:25.928461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928419    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy"
Nov 02 13:02:25.928461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928425    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy"
Nov 02 13:02:25.928587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928577    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="5202c1f9-f3a6-443e-ad9b-8254457cf1d9" containerName="maas-api"
Nov 02 13:02:25.928631 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.928588    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" containerName="istio-proxy"
Nov 02 13:02:25.933385 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.933366    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:25.955482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:25.955451    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:02:26.013979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.013945    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.014168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.014000    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.014168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.014024    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb2q\" (UniqueName: \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.014284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.014161    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.035250 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.035213    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0e422b-serving-cert\""
Nov 02 13:02:26.115787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.115741    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.116024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.115808    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.116024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.115848    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb2q\" (UniqueName: \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.116024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.115905    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.116572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.116548    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.118554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.118529    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.118658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.118587    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.141272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.141242    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb2q\" (UniqueName: \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\") pod \"maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.249471 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.249437    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:26.501876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.501761    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00b6cfc-4c1f-4911-b046-0438982d2379" path="/var/lib/kubelet/pods/b00b6cfc-4c1f-4911-b046-0438982d2379/volumes"
Nov 02 13:02:26.565638 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.565485    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:02:26.571682 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:02:26.571644    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0cedb3_905a_4ba2_ad3c_b751ef431ec1.slice/crio-db07f1ed31b221bbaf183f22bc347d9ad15314bd2175c0861f8cbc3d66490e73 WatchSource:0}: Error finding container db07f1ed31b221bbaf183f22bc347d9ad15314bd2175c0861f8cbc3d66490e73: Status 404 returned error can't find the container with id db07f1ed31b221bbaf183f22bc347d9ad15314bd2175c0861f8cbc3d66490e73
Nov 02 13:02:26.594023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:26.593989    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" event={"ID":"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1","Type":"ContainerStarted","Data":"db07f1ed31b221bbaf183f22bc347d9ad15314bd2175c0861f8cbc3d66490e73"}
Nov 02 13:02:27.648867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:27.644984    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" event={"ID":"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1","Type":"ContainerStarted","Data":"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"}
Nov 02 13:02:27.648867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:27.645904    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:27.693737 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:27.693684    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" podStartSLOduration=2.256051678 podStartE2EDuration="2.693664814s" totalImagesPullingTime="437.613136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:27.692579296 +0000 UTC m=+2145.341633255" watchObservedRunningTime="2026-11-02 13:02:27.693664814 +0000 UTC m=+2145.342718772"
Nov 02 13:02:34.670403 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:34.670366    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:02:39.760493 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:02:39.760455    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:02:42.547488 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.547444    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:02:42.588321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.588286    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:02:42.588669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.588645    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.605784 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.605751    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ef5efe-serving-cert\""
Nov 02 13:02:42.706282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.706245    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.706457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.706359    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.706457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.706418    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.706457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.706448    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjmh\" (UniqueName: \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.812162 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.812063    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.872469 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.872431    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.875995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.875961    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.898974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.898936    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjmh\" (UniqueName: \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.899597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.813291    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.902366 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.902341    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.902526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.902385    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:42.944384 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:42.944350    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjmh\" (UniqueName: \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\") pod \"maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:43.222935 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:43.222849    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:43.799247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:43.799212    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:02:43.893060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:43.892986    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" event={"ID":"87a0322b-c09b-43ef-974a-771a7a66fc47","Type":"ContainerStarted","Data":"bacdc48cf965c378b34ad4a923228fa7d2b00200a9912570cee0792b792d77d7"}
Nov 02 13:02:44.903098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:44.903060    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" event={"ID":"87a0322b-c09b-43ef-974a-771a7a66fc47","Type":"ContainerStarted","Data":"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"}
Nov 02 13:02:44.903940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:44.903915    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:44.997870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:44.997321    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" podStartSLOduration=2.567356705 podStartE2EDuration="2.997299624s" totalImagesPullingTime="429.942919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:44.982138132 +0000 UTC m=+2162.631192092" watchObservedRunningTime="2026-11-02 13:02:44.997299624 +0000 UTC m=+2162.646353582"
Nov 02 13:02:49.952681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.952585    4928 generic.go:356] "Generic (PLEG): container finished" podID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerID="d618487c66deff3093ffb75e7c9932be319202e9e0ae11309ddda238e898aa49" exitCode=0
Nov 02 13:02:49.952681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.952658    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerDied","Data":"d618487c66deff3093ffb75e7c9932be319202e9e0ae11309ddda238e898aa49"}
Nov 02 13:02:49.953327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.953221    4928 scope.go:122] "RemoveContainer" containerID="d618487c66deff3093ffb75e7c9932be319202e9e0ae11309ddda238e898aa49"
Nov 02 13:02:49.954894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.954867    4928 generic.go:356] "Generic (PLEG): container finished" podID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerID="3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095" exitCode=0
Nov 02 13:02:49.955004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.954930    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerDied","Data":"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"}
Nov 02 13:02:49.955302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.955289    4928 scope.go:122] "RemoveContainer" containerID="3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"
Nov 02 13:02:49.955455 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.955422    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:49.955535 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.955517    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:49.955586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.955571    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:49.957257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.957225    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:49.960836 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.957913    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:49.960836 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:49.958054    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:02:50.963847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:50.961937    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:02:50.968785 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:50.968746    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerStarted","Data":"c9b31636092d66928c1332b5168dbcab5255caf712d54757dbace4b046e5c8bc"}
Nov 02 13:02:50.971301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:50.971272    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerStarted","Data":"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"}
Nov 02 13:02:51.386198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.385882    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:02:51.386198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.385959    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:02:51.386602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.386576    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:51.386748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.386723    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:51.428849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.428797    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:51.429133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.429105    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:51.936852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:51.931015    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:02:52.209143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:52.209058    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:02:52.382851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:52.382569    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:52.382851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:52.382639    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:52.386851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:52.383335    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:52.386851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:52.383394    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:53.376255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:53.376214    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:53.376930 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:53.376897    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:53.382875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:53.382167    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:53.382875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:53.382199    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:54.381900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:54.379299    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:54.381900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:54.379362    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:54.401862 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:54.399256    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:54.401862 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:54.399321    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:55.375790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:55.375746    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:55.376196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:55.376148    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:55.381980 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:55.381948    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:55.382294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:55.382014    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:56.381237 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:56.381185    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 13:02:56.381459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:56.381257    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 13:02:56.381459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:56.381352    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 13:02:56.381459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:56.381407    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 13:02:57.381710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:57.381675    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:02:57.382254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:57.382043    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:02:57.382751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:57.382732    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:02:57.386378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:57.386353    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:02:57.387325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:02:57.387306    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:03:09.732812 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:03:09.732769    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:03:23.423920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:23.423548    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:03:23.423920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:23.423625    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:03:24.383705 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.383668    4928 generic.go:356] "Generic (PLEG): container finished" podID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerID="afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f" exitCode=0
Nov 02 13:03:24.383945 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.383737    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerDied","Data":"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"}
Nov 02 13:03:24.384221 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.384207    4928 scope.go:122] "RemoveContainer" containerID="afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"
Nov 02 13:03:24.385968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.385940    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:03:24.386052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.386009    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:03:24.386052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:24.386045    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:03:25.396309 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:25.396276    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerStarted","Data":"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"}
Nov 02 13:03:25.448850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:25.448797    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:03:25.452034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:25.452000    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:25.452281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:25.452252    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:26.448177 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:26.448135    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:26.449492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:26.449456    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:27.514314 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:27.510169    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:27.514314 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:27.510283    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:28.450966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:28.450926    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:28.451290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:28.451257    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:29.448199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:29.448159    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:29.448799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:29.448765    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:30.450231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:30.450191    4928 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 13:03:30.450883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:30.450809    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 13:03:31.451083 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:31.451046    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:03:31.451572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:31.451446    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:03:31.452243 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:03:31.452221    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:03:39.730924 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:03:39.730882    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:04:09.735109 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:04:09.735073    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:04:27.596526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.596487    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:04:27.601147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.601128    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.642968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.640148    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e3396f-serving-cert\""
Nov 02 13:04:27.652968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.652923    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:04:27.730796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.730754    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.731013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.730812    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.731013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.730920    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.731140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.731100    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c2w\" (UniqueName: \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.837850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.836512    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.837850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.836629    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.837850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.836663    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.837850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.836857    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c2w\" (UniqueName: \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.838254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.838137    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.846878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.846787    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.852925 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.847634    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.863979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.863419    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c2w\" (UniqueName: \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\") pod \"maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:27.915577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:27.915542    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:28.235021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:28.234808    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:04:28.239298 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:04:28.239269    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fff3afe_617b_43e3_b764_1b66db982d58.slice/crio-a0378560a74cf4467a48069752e1ca44eb2854a66383d8a31132ae086bab6f64 WatchSource:0}: Error finding container a0378560a74cf4467a48069752e1ca44eb2854a66383d8a31132ae086bab6f64: Status 404 returned error can't find the container with id a0378560a74cf4467a48069752e1ca44eb2854a66383d8a31132ae086bab6f64
Nov 02 13:04:29.110662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:29.110620    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" event={"ID":"2fff3afe-617b-43e3-b764-1b66db982d58","Type":"ContainerStarted","Data":"2dbe415836f97d9bb2778eb4eb569407f6c8de0d4333e6114e6b8a23135a2b26"}
Nov 02 13:04:29.111154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:29.110680    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" event={"ID":"2fff3afe-617b-43e3-b764-1b66db982d58","Type":"ContainerStarted","Data":"a0378560a74cf4467a48069752e1ca44eb2854a66383d8a31132ae086bab6f64"}
Nov 02 13:04:29.111154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:29.110850    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:29.156999 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:29.156937    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" podStartSLOduration=1.793475425 podStartE2EDuration="2.156917704s" totalImagesPullingTime="363.442279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:04:29.154780517 +0000 UTC m=+2266.803834478" watchObservedRunningTime="2026-11-02 13:04:29.156917704 +0000 UTC m=+2266.805971665"
Nov 02 13:04:35.124496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:04:35.124463    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:04:39.732501 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:04:39.732464    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:05:09.730986 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:05:09.730893    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:05:39.758972 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:05:39.758933    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:06:09.731843 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:06:09.731784    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:06:39.731212 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:06:39.731165    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:06:43.025196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:06:43.025167    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:06:43.029801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:06:43.029773    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:07:09.748323 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:07:09.748278    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:07:39.731217 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:07:39.731178    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:08:09.730603 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:08:09.730521    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:08:33.017312 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:33.017275    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 13:08:33.020015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:33.018956    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" containerID="cri-o://225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5" gracePeriod=30
Nov 02 13:08:37.515740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:37.515694    4928 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:08:37.516270 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:37.515770    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:08:38.324890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.324860    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 13:08:38.445002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.444960    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.445210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445039    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.445210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445163    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.445347 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445255    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:38.445419 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445356    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:38.445606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445578    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.445739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445656    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.445906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445868    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.446028 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.445945    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.446028 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.446020    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.446164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.446068    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\") pod \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\" (UID: \"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8\") "
Nov 02 13:08:38.446474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.446435    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:38.446575 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.446548    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:38.446769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.446743    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:38.447406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447382    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.447489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447413    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.447489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447427    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.447489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447439    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.447489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447455    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.448021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.447997    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:38.454034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.454008    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:38.454136 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.454032    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:08:38.454390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.454369    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd" pod "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" (UID: "7fcd3a0c-108b-4136-bd14-4bd73e25b4d8"). InnerVolumeSpecName "kube-api-access-hqmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:38.491130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.491102    4928 generic.go:356] "Generic (PLEG): container finished" podID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerID="225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5" exitCode=0
Nov 02 13:08:38.491283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.491223    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"
Nov 02 13:08:38.500176 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.500153    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" event={"ID":"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8","Type":"ContainerDied","Data":"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"}
Nov 02 13:08:38.500300 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.500196    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk" event={"ID":"7fcd3a0c-108b-4136-bd14-4bd73e25b4d8","Type":"ContainerDied","Data":"f01b62a28aa06bb5f423a2c65999ee5b0597afdca85374335d868d6e544c3d2d"}
Nov 02 13:08:38.500300 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.500229    4928 scope.go:122] "RemoveContainer" containerID="225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"
Nov 02 13:08:38.518744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.518719    4928 scope.go:122] "RemoveContainer" containerID="225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"
Nov 02 13:08:38.519109 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:08:38.519045    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5\": container with ID starting with 225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5 not found: ID does not exist" containerID="225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"
Nov 02 13:08:38.519109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.519071    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5"} err="failed to get container status \"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5\": rpc error: code = NotFound desc = could not find container \"225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5\": container with ID starting with 225bce25e01783a47ac8b3e88f54469e36589dfafd3b1cdc488d076dfe93b8a5 not found: ID does not exist"
Nov 02 13:08:38.548530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.548501    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.548530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.548531    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.548705 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.548547    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmkd\" (UniqueName: \"kubernetes.io/projected/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-kube-api-access-hqmkd\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.548705 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.548561    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.573854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.573783    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 13:08:38.599447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:38.599417    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kjvzk"]
Nov 02 13:08:39.731073 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:08:39.731039    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:08:39.791762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:39.791722    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 13:08:39.792070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:39.792040    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" containerID="cri-o://cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0" gracePeriod=30
Nov 02 13:08:40.501269 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:40.501230    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" path="/var/lib/kubelet/pods/7fcd3a0c-108b-4136-bd14-4bd73e25b4d8/volumes"
Nov 02 13:08:43.068784 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:43.068736    4928 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:08:43.069305 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:43.068876    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:08:45.094634 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.094607    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 13:08:45.147291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147249    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147452    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147488    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147534    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147590    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147620    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147682    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:45.147803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147696    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.147977 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.147955    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:45.148043 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.148011    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:45.148148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.148113    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:45.148148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.148133    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:45.148325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.148191    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.148370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.148323    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\") pod \"432729fb-e115-49e2-908a-db3f8309da85\" (UID: \"432729fb-e115-49e2-908a-db3f8309da85\") "
Nov 02 13:08:45.149146 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.149117    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.149273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.149149    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432729fb-e115-49e2-908a-db3f8309da85-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.149273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.149164    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.149273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.149178    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.149273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.149190    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.150244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.150221    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:08:45.150374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.150310    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "kube-api-access-p8g9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:45.150725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.150702    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:45.150798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.150731    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy" pod "432729fb-e115-49e2-908a-db3f8309da85" (UID: "432729fb-e115-49e2-908a-db3f8309da85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:45.250832 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.250784    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.251005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.250837    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432729fb-e115-49e2-908a-db3f8309da85-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.251005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.250855    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8g9j\" (UniqueName: \"kubernetes.io/projected/432729fb-e115-49e2-908a-db3f8309da85-kube-api-access-p8g9j\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.251005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.250868    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432729fb-e115-49e2-908a-db3f8309da85-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:08:45.566565 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.566510    4928 generic.go:356] "Generic (PLEG): container finished" podID="432729fb-e115-49e2-908a-db3f8309da85" containerID="cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0" exitCode=0
Nov 02 13:08:45.566923 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.566595    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" event={"ID":"432729fb-e115-49e2-908a-db3f8309da85","Type":"ContainerDied","Data":"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"}
Nov 02 13:08:45.567033 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.566999    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2" event={"ID":"432729fb-e115-49e2-908a-db3f8309da85","Type":"ContainerDied","Data":"d5098cbcd199053315a968dbddb14d41eb879c6078810431e0635b964bd3b66f"}
Nov 02 13:08:45.567033 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.566679    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"
Nov 02 13:08:45.567147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.567042    4928 scope.go:122] "RemoveContainer" containerID="cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"
Nov 02 13:08:45.580485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.580463    4928 scope.go:122] "RemoveContainer" containerID="cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"
Nov 02 13:08:45.580795 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:08:45.580775    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0\": container with ID starting with cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0 not found: ID does not exist" containerID="cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"
Nov 02 13:08:45.580884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.580808    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0"} err="failed to get container status \"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0\": rpc error: code = NotFound desc = could not find container \"cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0\": container with ID starting with cc3a8b063bcfdd16fc7211c1d95c9d37ef9d41bdce69c5351b884d67b69c18f0 not found: ID does not exist"
Nov 02 13:08:45.625290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.625258    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 13:08:45.639620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:45.639585    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-zfdc2"]
Nov 02 13:08:46.498732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:08:46.498696    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432729fb-e115-49e2-908a-db3f8309da85" path="/var/lib/kubelet/pods/432729fb-e115-49e2-908a-db3f8309da85/volumes"
Nov 02 13:09:09.731095 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:09:09.731054    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:09:27.149675 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.149590    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:09:27.150554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150528    4928 cpu_manager.go:414] "containerMap: removing container" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy"
Nov 02 13:09:27.150554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150554    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy"
Nov 02 13:09:27.150732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150604    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy"
Nov 02 13:09:27.150732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150612    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy"
Nov 02 13:09:27.150905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150764    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="432729fb-e115-49e2-908a-db3f8309da85" containerName="istio-proxy"
Nov 02 13:09:27.150905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.150786    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcd3a0c-108b-4136-bd14-4bd73e25b4d8" containerName="istio-proxy"
Nov 02 13:09:27.155255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.155229    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.183983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.183953    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:09:27.330796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.330755    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46l9x\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331020 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.330805    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331020 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.330879    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331022    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331191 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331142    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331222    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331366    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331411    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.331539 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.331523    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.432862 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.432756    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.432885    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.432929    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433008    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433059    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46l9x\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433098    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433134    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433232    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433263    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433351    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433592    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.433754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.433732    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.434226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.434194    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.434351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.434284    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.435931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.435898    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.436328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.436306    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.474843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.467521    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46l9x\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.486582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.475299    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jwzmj\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.486582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.475538    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:27.816938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.816902    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:09:27.818250 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:09:27.818221    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ece68d_f9b2_4b0b_b13b_620b4e2d2cc9.slice/crio-3180bfbaae1eff0791502db4086bf469a84928871a1b598d6e4eb5bcd46d0058 WatchSource:0}: Error finding container 3180bfbaae1eff0791502db4086bf469a84928871a1b598d6e4eb5bcd46d0058: Status 404 returned error can't find the container with id 3180bfbaae1eff0791502db4086bf469a84928871a1b598d6e4eb5bcd46d0058
Nov 02 13:09:27.821503 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.821465    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:27.821608 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.821538    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:27.821608 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.821580    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:27.966284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.965700    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" event={"ID":"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9","Type":"ContainerStarted","Data":"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"}
Nov 02 13:09:27.966284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:27.965756    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" event={"ID":"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9","Type":"ContainerStarted","Data":"3180bfbaae1eff0791502db4086bf469a84928871a1b598d6e4eb5bcd46d0058"}
Nov 02 13:09:28.020725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:28.020656    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podStartSLOduration=1.020635791 podStartE2EDuration="1.020635791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:28.013193343 +0000 UTC m=+2565.662247305" watchObservedRunningTime="2026-11-02 13:09:28.020635791 +0000 UTC m=+2565.669689750"
Nov 02 13:09:28.478888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:28.477708    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:28.479573 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:28.479549    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:28.479628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:28.479598    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:29.476926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:29.476885    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:29.477120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:29.476951    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:30.481685 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:30.481641    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:30.482257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:30.481721    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:31.477707 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:31.477663    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:31.477966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:31.477748    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:32.483910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:32.483221    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:32.483910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:32.483284    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:33.477695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:33.477649    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:33.477949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:33.477733    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:34.477729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:34.477678    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:34.478323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:34.477747    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:35.477909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:35.477778    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 02 13:09:35.477909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:35.477880    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 02 13:09:37.515047 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:37.514992    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jwzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:09:37.515606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:37.515086    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:09:37.561296 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:37.561260    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:37.561781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:37.561749    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:37.563430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:37.563408    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:09:39.806864 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:09:39.806786    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:09:42.208427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.208394    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:09:42.212730 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.212710    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.254229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.254194    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:09:42.260789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.260756    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:09:42.273291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.273258    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.312887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.305443    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:09:42.376035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.307656    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.376393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.376356    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.376558 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.376541    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.376702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.376683    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.376847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.376811    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.377001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.376983    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.377145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.377129    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.377311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.377285    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.377524 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.377505    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.377664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.377647    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.377872 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.377855    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.378130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378104    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.378337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378322    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.378468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378452    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwzc\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.378608 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378593    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.378742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378727    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.378943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.378926    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dztr\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.379109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.379092    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.485495 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.485429    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.486896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.486868    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.487126 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.487112    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.487244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.487231    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.487393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.487381    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.487492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.487479    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwzc\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.490885 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.490857    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.491249 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.491227    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.496273    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.496493    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.496929    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.498898    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.499900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.499267    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.500275 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.499924    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.500564 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.500541    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dztr\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.500747 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.500730    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.500947 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.500929    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.501353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.501330    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.501497 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.501481    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.504876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.504853    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.505432 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.505413    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.505649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.505619    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.506227 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.506205    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.506437 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.506417    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.506766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.501195    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.507100 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.505300    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.507303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.507277    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.509399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.507507    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.509399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.501139    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.509589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.509511    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.511868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.511548    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.511965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.511942    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.522629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.522604    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.522749 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.522612    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwzc\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.525541 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.525498    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dztr\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xxzqs\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.536389 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.536368    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-b2r5w\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.539397 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.539374    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:42.805958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.805871    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:42.945492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.936274    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:09:42.945492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.939752    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:42.945492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.939985    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:42.945492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:42.940046    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:43.142144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.142064    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:09:43.147426 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:09:43.147288    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ec0cc1_444b_4b80_90d3_104c4466f2d9.slice/crio-5ab876f010721135b9cd9c81a0b1cbdc9e88f2e5aedf5a5aa4bfda770f0599f7 WatchSource:0}: Error finding container 5ab876f010721135b9cd9c81a0b1cbdc9e88f2e5aedf5a5aa4bfda770f0599f7: Status 404 returned error can't find the container with id 5ab876f010721135b9cd9c81a0b1cbdc9e88f2e5aedf5a5aa4bfda770f0599f7
Nov 02 13:09:43.149948 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.149905    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:43.150218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.150000    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:43.150218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.150043    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:43.558852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.556646    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" event={"ID":"46ec0cc1-444b-4b80-90d3-104c4466f2d9","Type":"ContainerStarted","Data":"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"}
Nov 02 13:09:43.558852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.556712    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" event={"ID":"46ec0cc1-444b-4b80-90d3-104c4466f2d9","Type":"ContainerStarted","Data":"5ab876f010721135b9cd9c81a0b1cbdc9e88f2e5aedf5a5aa4bfda770f0599f7"}
Nov 02 13:09:43.574850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.561335    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" event={"ID":"254972ae-c46e-490a-8d7c-aa10b01b5698","Type":"ContainerStarted","Data":"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"}
Nov 02 13:09:43.574850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.561404    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" event={"ID":"254972ae-c46e-490a-8d7c-aa10b01b5698","Type":"ContainerStarted","Data":"1b85ca3c993be3d6f89e8386b68abf524f047039bbc4916e87b770b7f9569e3e"}
Nov 02 13:09:43.625968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.615360    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podStartSLOduration=1.615339334 podStartE2EDuration="1.615339334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:43.614787835 +0000 UTC m=+2581.263841794" watchObservedRunningTime="2026-11-02 13:09:43.615339334 +0000 UTC m=+2581.264393292"
Nov 02 13:09:43.703848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.703134    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podStartSLOduration=1.703111005 podStartE2EDuration="1.703111005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:43.678122851 +0000 UTC m=+2581.327176809" watchObservedRunningTime="2026-11-02 13:09:43.703111005 +0000 UTC m=+2581.352164965"
Nov 02 13:09:43.813048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:43.810457    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:44.548850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:44.547894    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:44.572850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:44.570509    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:44.572850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:44.570579    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:44.818069 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:44.817974    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:44.818355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:44.818323    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:45.553850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:45.552961    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:45.553850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:45.553052    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:45.806419 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:45.806329    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:45.806419 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:45.806399    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:46.559266 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:46.559229    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:46.559565 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:46.559530    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:46.811911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:46.809475    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:46.811911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:46.809541    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:47.556852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:47.556174    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:47.556852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:47.556236    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:47.833352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:47.832621    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:47.833352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:47.832708    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:48.548359 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:48.548322    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:48.549083 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:48.549049    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:48.806919 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:48.806712    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:48.806919 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:48.806785    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:49.543297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:49.543254    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:49.543942 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:49.543904    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:49.806492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:49.806394    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:49.806492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:49.806467    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:50.540278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:50.540231    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:50.540510 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:50.540304    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:50.806364 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:50.806267    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:50.806971 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:50.806932    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:51.540466 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:51.540425    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:51.540680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:51.540497    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:51.807056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:51.806960    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 13:09:51.807056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:51.807025    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 13:09:52.540204 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:52.540159    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 13:09:52.540429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:52.540232    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 13:09:53.850853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.850016    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xxzqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:09:53.850853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.850097    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:09:53.989275 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.987972    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:53.989275 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.989221    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:53.990587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.989704    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:09:53.990587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.989930    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:53.990587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.990261    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:53.991669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:53.991643    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:09:55.928850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.928347    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:09:55.940311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.940280    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:55.957622 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.957590    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"routing-overlay\""
Nov 02 13:09:55.974451 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.972639    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-148dac49c765\""
Nov 02 13:09:55.976566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.975923    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"kube-root-ca.crt\""
Nov 02 13:09:55.976734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.976053    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-90cda9\""
Nov 02 13:09:55.976910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.976148    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"openshift-service-ca.crt\""
Nov 02 13:09:55.977106 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:55.976289    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-e2e--4b89980e-dockercfg-lw85f\""
Nov 02 13:09:56.003924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.003866    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:09:56.039059 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.038983    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.039059 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.039043    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.039262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.039192    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54cx\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.039262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.039243    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.140791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.140752    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.141015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.140842    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.141144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.141116    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54cx\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.141253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.141233    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.141720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.141691    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.142129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.142109    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.143890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.143865    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.167627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.167600    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54cx\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.302352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.302317    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:56.610410 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:56.610378    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:09:56.611468 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:09:56.611433    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ba0da_34a1_4f2b_b862_307f410023bc.slice/crio-368547e27727df5b2ac29b8f7870b7d520296b117c6c472658212f078dac41b5 WatchSource:0}: Error finding container 368547e27727df5b2ac29b8f7870b7d520296b117c6c472658212f078dac41b5: Status 404 returned error can't find the container with id 368547e27727df5b2ac29b8f7870b7d520296b117c6c472658212f078dac41b5
Nov 02 13:09:57.039894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.039856    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" event={"ID":"067ba0da-34a1-4f2b-b862-307f410023bc","Type":"ContainerStarted","Data":"16e05ec9e3708c507a2587f987f8dd9c0b3dda9cb30229364fcf8c0399104eb8"}
Nov 02 13:09:57.040380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.039931    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" event={"ID":"067ba0da-34a1-4f2b-b862-307f410023bc","Type":"ContainerStarted","Data":"368547e27727df5b2ac29b8f7870b7d520296b117c6c472658212f078dac41b5"}
Nov 02 13:09:57.040380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.039965    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:09:57.102550 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.102499    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" podStartSLOduration=2.102484767 podStartE2EDuration="2.102484767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:57.100423895 +0000 UTC m=+2594.749477852" watchObservedRunningTime="2026-11-02 13:09:57.102484767 +0000 UTC m=+2594.751538724"
Nov 02 13:09:57.541871 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.541817    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:09:57.546485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.546462    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.581093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.581060    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e7561a-openshift-default-dockercfg-87qsn\""
Nov 02 13:09:57.586651 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.586618    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:09:57.665247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665208    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665425 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665296    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665419    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665506 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665491    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665608    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665667 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665653    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hqv\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665755    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665855 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665814    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.665899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.665885    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.767663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.767619    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.767884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.767700    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.767884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.767772    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.767884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.767857    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.767928    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768015    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768066    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hqv\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768130    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768179    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768228    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768389 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768364    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768415    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768497 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768479    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.768710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.768691    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.770344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.770320    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.770756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.770715    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.793402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.793340    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.793518 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.793429    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hqv\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:57.861901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:57.861864    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:58.198005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:58.197973    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:09:58.198716 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:09:58.198688    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b70a8b_ad95_4121_8adc_39f7283007ab.slice/crio-996196c7efcce88f04027d20b45c3c0c92d4c4a700023e983b6936a1994e2b9c WatchSource:0}: Error finding container 996196c7efcce88f04027d20b45c3c0c92d4c4a700023e983b6936a1994e2b9c: Status 404 returned error can't find the container with id 996196c7efcce88f04027d20b45c3c0c92d4c4a700023e983b6936a1994e2b9c
Nov 02 13:09:58.202011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:58.201966    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:58.202135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:58.202056    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:58.202135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:58.202098    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:09:59.061315 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.061280    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" event={"ID":"f0b70a8b-ad95-4121-8adc-39f7283007ab","Type":"ContainerStarted","Data":"648c717d892bc54c93c9b3f321a1fa78e3dc6ebd51e1d4da4ee519266304d0bd"}
Nov 02 13:09:59.061484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.061339    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" event={"ID":"f0b70a8b-ad95-4121-8adc-39f7283007ab","Type":"ContainerStarted","Data":"996196c7efcce88f04027d20b45c3c0c92d4c4a700023e983b6936a1994e2b9c"}
Nov 02 13:09:59.117118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.117053    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podStartSLOduration=2.117036067 podStartE2EDuration="2.117036067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:59.1150516 +0000 UTC m=+2596.764105551" watchObservedRunningTime="2026-11-02 13:09:59.117036067 +0000 UTC m=+2596.766090026"
Nov 02 13:09:59.862305 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.862264    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:09:59.864123 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.864089    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 13:09:59.864255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:09:59.864151    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 13:10:00.862231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:00.862190    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 13:10:00.862433 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:00.862285    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 13:10:01.863192 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:01.863156    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 13:10:01.863630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:01.863207    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 13:10:01.973240 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:01.973200    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:10:01.978296 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:01.978273    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.005680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.005130    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-90cda9-openshift-default-dockercfg-xm76t\""
Nov 02 13:10:02.011172 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.011134    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:10:02.071968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.071908    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.071968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.071955    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.071978    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072050    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072144    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x994\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072162    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072200    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072523 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072228    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.072523 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.072241    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.173897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.173749    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x994\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.173897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.173801    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.173897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.173854    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.173897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.173895    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.173916    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.174020    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.174044    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.174060    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.174140    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.174866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.174844    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.175642 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.175357    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.176254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.176228    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.178869 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.178844    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.179071 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.179053    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.179382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.179358    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.179670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.179651    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.200633 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.200249    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.200764 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.200647    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x994\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.294324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.294285    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:02.663620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.663586    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:10:02.669483 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.669448    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:10:02.669603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.669526    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:10:02.669603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.669573    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:10:02.866254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.865492    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 13:10:02.866254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:02.865579    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 13:10:03.100848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.100234    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" event={"ID":"b7f57607-fb21-452a-8621-d92cda863242","Type":"ContainerStarted","Data":"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"}
Nov 02 13:10:03.100848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.100316    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" event={"ID":"b7f57607-fb21-452a-8621-d92cda863242","Type":"ContainerStarted","Data":"ad84ff1149b339c36ad2d8b0abd79b850435456e9d4836eed70b387140ecc558"}
Nov 02 13:10:03.295573 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.295531    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:03.297247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.297212    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:03.297399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.297258    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:03.865848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.865739    4928 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 13:10:03.866053 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:03.865806    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 13:10:04.295493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.295454    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:04.296031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.295517    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:04.867618 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.867584    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:10:04.868118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.868089    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:10:04.868938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.868916    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:10:04.966055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:04.965191    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podStartSLOduration=3.965176457 podStartE2EDuration="3.965176457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:03.15703854 +0000 UTC m=+2600.806092502" watchObservedRunningTime="2026-11-02 13:10:04.965176457 +0000 UTC m=+2602.614230420"
Nov 02 13:10:05.300431 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:05.300398    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:05.334034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:05.333986    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:06.295766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.295726    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:06.295982 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.295800    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:06.614744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.614655    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:10:06.619545 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.619522    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.658849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.655032    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-08daf0d4ee58\""
Nov 02 13:10:06.658849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.658729    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:10:06.672847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.672792    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.673132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.672921    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.673132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.672997    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.673364 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.673209    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65f55\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.774267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.774234    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.774267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.774274    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.774547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.774336    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.774547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.774379    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65f55\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.775108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.775078    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.775264 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.775239    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.777132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.777106    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.799372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.799342    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65f55\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:06.980901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:06.973327    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:07.295604 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:07.295562    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:07.295775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:07.295642    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:07.373680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:07.373648    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:10:07.373774 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:10:07.373704    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64567bb_8703_4dba_ae12_50436554814d.slice/crio-9bab23372eea775374426567215b7be4801e25039231440722ee2fed3c660cfb WatchSource:0}: Error finding container 9bab23372eea775374426567215b7be4801e25039231440722ee2fed3c660cfb: Status 404 returned error can't find the container with id 9bab23372eea775374426567215b7be4801e25039231440722ee2fed3c660cfb
Nov 02 13:10:08.057660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.057624    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:10:08.195875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.195593    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" event={"ID":"e64567bb-8703-4dba-ae12-50436554814d","Type":"ContainerStarted","Data":"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"}
Nov 02 13:10:08.195875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.195673    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" event={"ID":"e64567bb-8703-4dba-ae12-50436554814d","Type":"ContainerStarted","Data":"9bab23372eea775374426567215b7be4801e25039231440722ee2fed3c660cfb"}
Nov 02 13:10:08.195875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.195723    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:08.264477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.264415    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" podStartSLOduration=2.264399336 podStartE2EDuration="2.264399336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:08.263682429 +0000 UTC m=+2605.912736388" watchObservedRunningTime="2026-11-02 13:10:08.264399336 +0000 UTC m=+2605.913453293"
Nov 02 13:10:08.295168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.295131    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:08.295359 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:08.295207    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:09.295297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:09.295260    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 13:10:09.295894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:09.295318    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 13:10:09.730709 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:10:09.730624    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:10:10.300321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:10.300289    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:10.300813 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:10.300689    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:10.301333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:10.301311    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:10:19.209895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:19.209865    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:10:19.287343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:19.287307    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:10:24.288851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:24.288104    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" podUID="067ba0da-34a1-4f2b-b862-307f410023bc" containerName="payload-processing" containerID="cri-o://16e05ec9e3708c507a2587f987f8dd9c0b3dda9cb30229364fcf8c0399104eb8" gracePeriod=25
Nov 02 13:10:24.846929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:24.844649    4928 generic.go:356] "Generic (PLEG): container finished" podID="067ba0da-34a1-4f2b-b862-307f410023bc" containerID="16e05ec9e3708c507a2587f987f8dd9c0b3dda9cb30229364fcf8c0399104eb8" exitCode=0
Nov 02 13:10:24.884847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:24.878422    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" event={"ID":"067ba0da-34a1-4f2b-b862-307f410023bc","Type":"ContainerDied","Data":"16e05ec9e3708c507a2587f987f8dd9c0b3dda9cb30229364fcf8c0399104eb8"}
Nov 02 13:10:25.041850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.037907    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:10:25.041850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.038015    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:25.141912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.139043    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:25.328966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.324145    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:10:25.354976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.354539    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\") pod \"067ba0da-34a1-4f2b-b862-307f410023bc\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") "
Nov 02 13:10:25.354976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.354631    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\") pod \"067ba0da-34a1-4f2b-b862-307f410023bc\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") "
Nov 02 13:10:25.358731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.358698    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx" pod "067ba0da-34a1-4f2b-b862-307f410023bc" (UID: "067ba0da-34a1-4f2b-b862-307f410023bc"). InnerVolumeSpecName "kube-api-access-l54cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:25.359045 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.359001    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials" pod "067ba0da-34a1-4f2b-b862-307f410023bc" (UID: "067ba0da-34a1-4f2b-b862-307f410023bc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:25.456756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.456646    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\") pod \"067ba0da-34a1-4f2b-b862-307f410023bc\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") "
Nov 02 13:10:25.457137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.457119    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\") pod \"067ba0da-34a1-4f2b-b862-307f410023bc\" (UID: \"067ba0da-34a1-4f2b-b862-307f410023bc\") "
Nov 02 13:10:25.458006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.457988    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54cx\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-kube-api-access-l54cx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.458145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.458134    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/067ba0da-34a1-4f2b-b862-307f410023bc-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.458634 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.458612    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume" pod "067ba0da-34a1-4f2b-b862-307f410023bc" (UID: "067ba0da-34a1-4f2b-b862-307f410023bc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:25.458760 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.458670    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume" pod "067ba0da-34a1-4f2b-b862-307f410023bc" (UID: "067ba0da-34a1-4f2b-b862-307f410023bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:25.559375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.559335    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.559375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.559373    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067ba0da-34a1-4f2b-b862-307f410023bc-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.855330 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.855295    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"
Nov 02 13:10:25.855547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.855328    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8" event={"ID":"067ba0da-34a1-4f2b-b862-307f410023bc","Type":"ContainerDied","Data":"368547e27727df5b2ac29b8f7870b7d520296b117c6c472658212f078dac41b5"}
Nov 02 13:10:25.855547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.855401    4928 scope.go:122] "RemoveContainer" containerID="16e05ec9e3708c507a2587f987f8dd9c0b3dda9cb30229364fcf8c0399104eb8"
Nov 02 13:10:25.912954 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.912919    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:10:25.926373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:25.926334    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6c67rz8"]
Nov 02 13:10:26.498819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:26.498781    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067ba0da-34a1-4f2b-b862-307f410023bc" path="/var/lib/kubelet/pods/067ba0da-34a1-4f2b-b862-307f410023bc/volumes"
Nov 02 13:10:34.194778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.194740    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:10:34.195518 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.195501    4928 cpu_manager.go:414] "containerMap: removing container" podUID="067ba0da-34a1-4f2b-b862-307f410023bc" containerName="payload-processing"
Nov 02 13:10:34.195518 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.195519    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067ba0da-34a1-4f2b-b862-307f410023bc" containerName="payload-processing"
Nov 02 13:10:34.195645 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.195627    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="067ba0da-34a1-4f2b-b862-307f410023bc" containerName="payload-processing"
Nov 02 13:10:34.200236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.200207    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.218351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.218323    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:10:34.226720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.226689    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:10:34.226938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.226731    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8e7fc4b63cd8\""
Nov 02 13:10:34.227062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.226772    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-90aa2c2f03a4\""
Nov 02 13:10:34.227062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.226920    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:10:34.227250 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.226934    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:10:34.229729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.229703    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:10:34.229966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.229949    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vjcqq\""
Nov 02 13:10:34.322809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.322776    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.323024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.322864    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.323078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.323033    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnfs\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.323141 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.323088    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.424505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.424462    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnfs\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.424691 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.424557    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.424741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.424690    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.424801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.424736    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.425619 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.425587    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.425619 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.425607    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.427284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.427265    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.449511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.449448    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnfs\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\") pod \"payload-processing-external-model-57fd545999-tmffg\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") " pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.515408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.515371    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:34.835509 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:34.835463    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:10:34.836976 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:10:34.836946    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd6a326_3419_4597_8938_c075f194086f.slice/crio-1d47b4797ea966b3135594419c11f0b119560faa1f20a60b872eb0a22227f213 WatchSource:0}: Error finding container 1d47b4797ea966b3135594419c11f0b119560faa1f20a60b872eb0a22227f213: Status 404 returned error can't find the container with id 1d47b4797ea966b3135594419c11f0b119560faa1f20a60b872eb0a22227f213
Nov 02 13:10:35.009228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:35.009186    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" event={"ID":"dfd6a326-3419-4597-8938-c075f194086f","Type":"ContainerStarted","Data":"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"}
Nov 02 13:10:35.009426 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:35.009287    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" event={"ID":"dfd6a326-3419-4597-8938-c075f194086f","Type":"ContainerStarted","Data":"1d47b4797ea966b3135594419c11f0b119560faa1f20a60b872eb0a22227f213"}
Nov 02 13:10:35.009502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:35.009422    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:35.055602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:35.055548    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" podStartSLOduration=1.055532885 podStartE2EDuration="1.055532885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:35.052775934 +0000 UTC m=+2632.701829900" watchObservedRunningTime="2026-11-02 13:10:35.055532885 +0000 UTC m=+2632.704586872"
Nov 02 13:10:39.732119 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:10:39.732075    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:10:46.021346 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:46.021310    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:10:57.968906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:57.965035    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:58.027572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:58.027525    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:58.070731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:10:58.069969    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:09.731019 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:09.730978    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:11:15.092500 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.092446    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:11:15.111082 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.111051    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.149886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.149789    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bbb7d0-openshift-default-dockercfg-wzl2h\""
Nov 02 13:11:15.151697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.151667    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184639    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184690    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184715    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184746    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184799    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184932    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.184974    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.185024    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.199173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.185160    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.381712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.381626    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384029    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384215    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384339    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384471 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384458    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384597    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.384877 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384814    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.385121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.384999    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.385121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.385039    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.385121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.385097    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.387273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.387247    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.390196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.390167    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.390480 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.390452    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.390743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.390715    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.390921 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.390903    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.391010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.390988    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.417992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.416169    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.417992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.416206    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\") pod \"e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") " pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.577845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.577797    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:15.936017 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:11:15.935987    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e41d50_b9be_47bb_a01f_5f0b631596bb.slice/crio-0ee5ee02e69776955dcaaf3dd29da1f32edece34fc285a42dafe34e0c0c5a816 WatchSource:0}: Error finding container 0ee5ee02e69776955dcaaf3dd29da1f32edece34fc285a42dafe34e0c0c5a816: Status 404 returned error can't find the container with id 0ee5ee02e69776955dcaaf3dd29da1f32edece34fc285a42dafe34e0c0c5a816
Nov 02 13:11:15.938618 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.938591    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:11:15.939606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.939566    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:11:15.939714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.939655    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:11:15.939714 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:15.939705    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:11:16.892379 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:16.892339    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerStarted","Data":"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"}
Nov 02 13:11:16.892885 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:16.892399    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerStarted","Data":"0ee5ee02e69776955dcaaf3dd29da1f32edece34fc285a42dafe34e0c0c5a816"}
Nov 02 13:11:16.969420 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:16.969359    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podStartSLOduration=1.96934535 podStartE2EDuration="1.96934535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:16.953174145 +0000 UTC m=+2674.602228103" watchObservedRunningTime="2026-11-02 13:11:16.96934535 +0000 UTC m=+2674.618399308"
Nov 02 13:11:17.578905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:17.578866    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:17.585469 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:17.585443    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:17.901856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:17.901732    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:17.902765 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:17.902743    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:11:25.999813 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:25.999776    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:11:26.005852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.005813    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.035981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.035949    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"payload-processing-external-model-e2e--528e896e-dockercfg-rxfvj\""
Nov 02 13:11:26.036161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.035955    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"openshift-service-ca.crt\""
Nov 02 13:11:26.036271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.036008    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3396f\""
Nov 02 13:11:26.036345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.036032    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"agc-cred-f37388fe452e\""
Nov 02 13:11:26.036414 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.036074    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"routing-overlay\""
Nov 02 13:11:26.036481 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.036161    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"kube-root-ca.crt\""
Nov 02 13:11:26.043710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.043680    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:11:26.186849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.185586    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58n24\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.186849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.185736    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.186849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.185920    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.186849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.185984    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.290896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.288795    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58n24\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.290896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.288963    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.290896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.289140    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.290896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.289205    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.290896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.290243    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.294843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.291358    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.342847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.329758    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.350849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.348386    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58n24\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.624107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.624021    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:26.943427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:26.943395    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:11:26.944923 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:11:26.944888    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fecb316_2d2c_4894_86e0_6f1ec1a7fd68.slice/crio-80f4659a9eaf1ecbb1edd291c5c106bc0745de93ed48c82de0dd215abd5eac03 WatchSource:0}: Error finding container 80f4659a9eaf1ecbb1edd291c5c106bc0745de93ed48c82de0dd215abd5eac03: Status 404 returned error can't find the container with id 80f4659a9eaf1ecbb1edd291c5c106bc0745de93ed48c82de0dd215abd5eac03
Nov 02 13:11:27.014234 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:27.014203    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" event={"ID":"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68","Type":"ContainerStarted","Data":"80f4659a9eaf1ecbb1edd291c5c106bc0745de93ed48c82de0dd215abd5eac03"}
Nov 02 13:11:28.023652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:28.023614    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" event={"ID":"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68","Type":"ContainerStarted","Data":"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"}
Nov 02 13:11:28.024133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:28.023702    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:28.070646 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:28.070589    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" podStartSLOduration=3.070576085 podStartE2EDuration="3.070576085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:28.068903234 +0000 UTC m=+2685.717957193" watchObservedRunningTime="2026-11-02 13:11:28.070576085 +0000 UTC m=+2685.719630043"
Nov 02 13:11:39.036762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:39.036729    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:11:39.730671 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:39.730627    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:11:43.091936 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:43.091890    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:11:43.096513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:43.096486    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:11:51.659053 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:51.659019    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:51.659743 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:51.659726    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:51.660666 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:51.660649    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:11:52.160626939 +0000 UTC m=+2709.809680881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:52.165034 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:52.165001    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:52.165034 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:52.165035    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:52.165323 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:52.165096    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:11:53.16507848 +0000 UTC m=+2710.814132433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:53.193848 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:53.193630    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:53.193848 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:53.193674    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:53.193848 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:53.193737    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:11:55.193717548 +0000 UTC m=+2712.842771496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:54.006992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.006928    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:54.031123 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.028904    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:54.165077 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.152169    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:54.406014 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.405580    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:54.451202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.451157    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:11:54.713703 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:11:54.713591    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:11:55.195444 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:55.195406    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:55.195444 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:55.195442    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:55.195668 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:55.195499    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:11:59.195484347 +0000 UTC m=+2716.844538283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:59.260792 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:59.260760    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:59.260792 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:59.260791    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:11:59.261263 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:11:59.260854    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:12:07.260839645 +0000 UTC m=+2724.909893597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:07.322127 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:07.322083    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:07.322127 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:07.322120    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:07.322706 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:07.322172    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:12:23.322157478 +0000 UTC m=+2740.971211413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:09.731422 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:09.731383    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:12:23.378618 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:23.378574    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:23.378618 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:23.378616    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:23.379196 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:23.378694    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:12:55.378674016 +0000 UTC m=+2773.027727969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:39.971620 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:39.971573    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:12:40.005002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.004946    4928 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.017432 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.017381    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.048078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.048024    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-b2r5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:12:40.048432 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.048391    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.101037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.100974    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.134445 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.134397    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.439967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.439915    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:40.482847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.482792    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 13:12:40.483225 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.483102    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:12:40.716209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.716097    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:12:40.963960 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:40.963910    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 13:12:41.006973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.006917    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 13:12:41.116893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.113522    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 13:12:41.116893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.114320    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 13:12:41.135103 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.135063    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 02 13:12:41.136746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.136719    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 13:12:41.497433 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.497374    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:41.568851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.564070    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 13:12:41.568851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.497730    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:41.570202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.498544    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5" podUID="d3e5f98e-cab4-4a6b-8884-efda80d420ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:12:41.572976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.570459    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 13:12:41.572976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.570522    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 13:12:41.652993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.625419    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 13:12:41.734555 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.734509    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:12:41.764579 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.764471    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 13:12:41.981352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.981300    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 13:12:42.047004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:41.994044    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 13:12:42.120663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.105069    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 02 13:12:42.120663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.116075    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 13:12:42.120663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.116657    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 13:12:42.120663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.116713    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 13:12:42.193224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.144952    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 13:12:42.193224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.185381    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 13:12:42.200034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.199976    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 13:12:42.214017 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.213958    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 13:12:42.214913 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:42.214888    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5"
Nov 02 13:12:47.903598 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:47.903555    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:47.904134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:47.903628    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:48.244669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.244627    4928 generic.go:356] "Generic (PLEG): container finished" podID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerID="e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3" exitCode=0
Nov 02 13:12:48.244922 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.244763    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerDied","Data":"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"}
Nov 02 13:12:48.245306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.245283    4928 scope.go:122] "RemoveContainer" containerID="e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"
Nov 02 13:12:48.247280 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.247246    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:48.247399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.247321    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:48.247399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:48.247355    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:49.273974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:49.273928    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerStarted","Data":"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"}
Nov 02 13:12:49.578745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:49.578644    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:12:49.578958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:49.578889    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:49.578958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:49.578942    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:50.578638 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:50.578588    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:50.579174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:50.578657    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:51.578851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:51.578803    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:51.579400 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:51.578866    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:52.260853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.256093    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:12:52.285814 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.285779    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.286268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.286242    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:12:52.303196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.303169    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9139ec-openshift-default-dockercfg-4ltv6\""
Nov 02 13:12:52.459394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.459342    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.459873 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.459813    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.460121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.460097    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.475850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.460275    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.476138 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.476114    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.483849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.480113    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.483849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.480200    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.483849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.480278    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.483849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.480332    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dc44\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.579480    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.579557    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.585892    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.585971    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586007    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dc44\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586149    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586189    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586253    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586335    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.586505    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.588926    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.589203    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.613901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.589434    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.617262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.616421    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.617262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.617124    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.711861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.704411    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.711861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.709620    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.712126 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.711969    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dc44\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.715539 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.715480    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:52.908859 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:52.908759    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:53.314183 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.314145    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:12:53.317400 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:12:53.317367    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6dfe07_9efb_4437_a111_d30aabccf018.slice/crio-149464ffe756803092a382c7b1a507fe93456a8ddd242c6b16058d6658da5b32 WatchSource:0}: Error finding container 149464ffe756803092a382c7b1a507fe93456a8ddd242c6b16058d6658da5b32: Status 404 returned error can't find the container with id 149464ffe756803092a382c7b1a507fe93456a8ddd242c6b16058d6658da5b32
Nov 02 13:12:53.320399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.320363    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:53.320496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.320436    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:53.320496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.320468    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:12:53.578447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.578359    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:53.578447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:53.578413    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:54.350851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.350416    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerStarted","Data":"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"}
Nov 02 13:12:54.350851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.350494    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerStarted","Data":"149464ffe756803092a382c7b1a507fe93456a8ddd242c6b16058d6658da5b32"}
Nov 02 13:12:54.418517 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.418458    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podStartSLOduration=2.418443513 podStartE2EDuration="2.418443513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:54.411435162 +0000 UTC m=+2772.060489132" watchObservedRunningTime="2026-11-02 13:12:54.418443513 +0000 UTC m=+2772.067497470"
Nov 02 13:12:54.581882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.581233    4928 patch_prober.go:29] interesting pod/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 13:12:54.581882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.581297    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 13:12:54.914850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.910136    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:54.939700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:54.939168    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:55.313272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.313236    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:12:55.364332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.364298    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:55.365299 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.365276    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:12:55.380982 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:55.380958    4928 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8e7fc4b63cd8: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:55.380982 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:55.380989    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57fd545999-tmffg: secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:55.381129 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:12:55.381053    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials podName:dfd6a326-3419-4597-8938-c075f194086f nodeName:}" failed. No retries permitted until 2026-11-02 13:13:59.381035988 +0000 UTC m=+2837.030089945 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials") pod "payload-processing-external-model-57fd545999-tmffg" (UID: "dfd6a326-3419-4597-8938-c075f194086f") : secret "agc-cred-8e7fc4b63cd8" not found
Nov 02 13:12:55.584356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.584272    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:12:55.584665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.584638    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:12:55.585546 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:55.585526    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:12:57.171805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:12:57.171772    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:13:00.367017 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.366954    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" podUID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" containerName="payload-processing" containerID="cri-o://8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6" gracePeriod=25
Nov 02 13:13:00.702248 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.702220    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:13:00.732753 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.732723    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\") pod \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") "
Nov 02 13:13:00.732924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.732858    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\") pod \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") "
Nov 02 13:13:00.733000 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.732983    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\") pod \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") "
Nov 02 13:13:00.733057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.733026    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\") pod \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\" (UID: \"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68\") "
Nov 02 13:13:00.733430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.733398    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume" pod "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" (UID: "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:00.733430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.733414    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume" pod "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" (UID: "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:00.733926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.733904    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:00.734012 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.733930    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:00.735099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.735073    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials" pod "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" (UID: "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:00.735212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.735111    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24" pod "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" (UID: "8fecb316-2d2c-4894-86e0-6f1ec1a7fd68"). InnerVolumeSpecName "kube-api-access-58n24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:00.835457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.835420    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58n24\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-kube-api-access-58n24\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:00.835457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:00.835452    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:01.482587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.482551    4928 generic.go:356] "Generic (PLEG): container finished" podID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" containerID="8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6" exitCode=0
Nov 02 13:13:01.483050 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.482598    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"
Nov 02 13:13:01.483050 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.482632    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" event={"ID":"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68","Type":"ContainerDied","Data":"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"}
Nov 02 13:13:01.483050 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.482690    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw" event={"ID":"8fecb316-2d2c-4894-86e0-6f1ec1a7fd68","Type":"ContainerDied","Data":"80f4659a9eaf1ecbb1edd291c5c106bc0745de93ed48c82de0dd215abd5eac03"}
Nov 02 13:13:01.483050 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.482720    4928 scope.go:122] "RemoveContainer" containerID="8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"
Nov 02 13:13:01.495153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.495124    4928 scope.go:122] "RemoveContainer" containerID="8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"
Nov 02 13:13:01.495491 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:01.495467    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6\": container with ID starting with 8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6 not found: ID does not exist" containerID="8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"
Nov 02 13:13:01.495577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.495499    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6"} err="failed to get container status \"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6\": rpc error: code = NotFound desc = could not find container \"8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6\": container with ID starting with 8363a7e0a9b9c9e9e0cd11c33855508f1bbfea5f8e3087e3b8051753fe2568d6 not found: ID does not exist"
Nov 02 13:13:01.551567 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.551536    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:13:01.563806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:01.563777    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-7cd9vbkrw"]
Nov 02 13:13:02.172280 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.172234    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" podUID="dfd6a326-3419-4597-8938-c075f194086f" containerName="payload-processing" containerID="cri-o://a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46" gracePeriod=25
Nov 02 13:13:02.458750 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.458720    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:13:02.491739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.491708    4928 generic.go:356] "Generic (PLEG): container finished" podID="dfd6a326-3419-4597-8938-c075f194086f" containerID="a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46" exitCode=0
Nov 02 13:13:02.492119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.491803    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg"
Nov 02 13:13:02.499392 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.499366    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" path="/var/lib/kubelet/pods/8fecb316-2d2c-4894-86e0-6f1ec1a7fd68/volumes"
Nov 02 13:13:02.501911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.501877    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" event={"ID":"dfd6a326-3419-4597-8938-c075f194086f","Type":"ContainerDied","Data":"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"}
Nov 02 13:13:02.502032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.501932    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fd545999-tmffg" event={"ID":"dfd6a326-3419-4597-8938-c075f194086f","Type":"ContainerDied","Data":"1d47b4797ea966b3135594419c11f0b119560faa1f20a60b872eb0a22227f213"}
Nov 02 13:13:02.502032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.501957    4928 scope.go:122] "RemoveContainer" containerID="a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"
Nov 02 13:13:02.512388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.512367    4928 scope.go:122] "RemoveContainer" containerID="a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"
Nov 02 13:13:02.512685 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:02.512652    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46\": container with ID starting with a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46 not found: ID does not exist" containerID="a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"
Nov 02 13:13:02.512769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.512684    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46"} err="failed to get container status \"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46\": rpc error: code = NotFound desc = could not find container \"a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46\": container with ID starting with a974d1108c5511d46c938f965113c63c745c5285517c30f695077eb4728baa46 not found: ID does not exist"
Nov 02 13:13:02.572662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.572622    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\") pod \"dfd6a326-3419-4597-8938-c075f194086f\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") "
Nov 02 13:13:02.572864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.572779    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\") pod \"dfd6a326-3419-4597-8938-c075f194086f\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") "
Nov 02 13:13:02.572864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.572842    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\") pod \"dfd6a326-3419-4597-8938-c075f194086f\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") "
Nov 02 13:13:02.573006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.572887    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\") pod \"dfd6a326-3419-4597-8938-c075f194086f\" (UID: \"dfd6a326-3419-4597-8938-c075f194086f\") "
Nov 02 13:13:02.573150 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.573118    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume" pod "dfd6a326-3419-4597-8938-c075f194086f" (UID: "dfd6a326-3419-4597-8938-c075f194086f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:02.573477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.573413    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume" pod "dfd6a326-3419-4597-8938-c075f194086f" (UID: "dfd6a326-3419-4597-8938-c075f194086f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:02.573891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.573871    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.573970 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.573893    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfd6a326-3419-4597-8938-c075f194086f-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.575376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.575351    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials" pod "dfd6a326-3419-4597-8938-c075f194086f" (UID: "dfd6a326-3419-4597-8938-c075f194086f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:02.575654 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.575634    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs" pod "dfd6a326-3419-4597-8938-c075f194086f" (UID: "dfd6a326-3419-4597-8938-c075f194086f"). InnerVolumeSpecName "kube-api-access-djnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:02.675796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.675718    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.675796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.675746    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnfs\" (UniqueName: \"kubernetes.io/projected/dfd6a326-3419-4597-8938-c075f194086f-kube-api-access-djnfs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.846111 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.846074    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:13:02.850122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:02.850090    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fd545999-tmffg"]
Nov 02 13:13:04.500966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:04.500920    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd6a326-3419-4597-8938-c075f194086f" path="/var/lib/kubelet/pods/dfd6a326-3419-4597-8938-c075f194086f/volumes"
Nov 02 13:13:05.705098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705058    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:13:05.705861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705811    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" containerName="payload-processing"
Nov 02 13:13:05.705861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705842    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" containerName="payload-processing"
Nov 02 13:13:05.705861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705856    4928 cpu_manager.go:414] "containerMap: removing container" podUID="dfd6a326-3419-4597-8938-c075f194086f" containerName="payload-processing"
Nov 02 13:13:05.705861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705862    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd6a326-3419-4597-8938-c075f194086f" containerName="payload-processing"
Nov 02 13:13:05.706074 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705980    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fecb316-2d2c-4894-86e0-6f1ec1a7fd68" containerName="payload-processing"
Nov 02 13:13:05.706074 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.705992    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd6a326-3419-4597-8938-c075f194086f" containerName="payload-processing"
Nov 02 13:13:05.711378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.711356    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:05.730244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.730216    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:13:05.737490 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.737460    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6563ed\""
Nov 02 13:13:05.737664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.737577    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6563ed-dockercfg-c7zm7\""
Nov 02 13:13:05.898849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.868905    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:13:05.898849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.879592    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:05.909863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.909409    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:13:05.945302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.931760    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4g7\" (UniqueName: \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:05.948848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:05.946374    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.048668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.048597    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4g7\" (UniqueName: \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.049352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.049323    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.049516 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.049400    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.049516 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.049496    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs7h\" (UniqueName: \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.050759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.050491    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.081745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.081712    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4g7\" (UniqueName: \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\") pod \"payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.151314 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.151278    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.151523 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.151404    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs7h\" (UniqueName: \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.152105 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.152075    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.176864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.176790    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs7h\" (UniqueName: \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\") pod \"payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.293711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.293682    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:06.327810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.327780    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:06.614728 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:13:06.614699    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f79307_ad18_4efa_bfe4_13d23e590c56.slice/crio-639e1464fdd85f4101e7796685136dfaff7f9c344071537c14d5474203f11f87 WatchSource:0}: Error finding container 639e1464fdd85f4101e7796685136dfaff7f9c344071537c14d5474203f11f87: Status 404 returned error can't find the container with id 639e1464fdd85f4101e7796685136dfaff7f9c344071537c14d5474203f11f87
Nov 02 13:13:06.615277 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.615254    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:13:06.661950 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:13:06.661912    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcab597_c6c0_4577_b9fa_a1e20c567b56.slice/crio-4fb3530b07cf3d807292acddded7eef0a7e69f32e5efdfd591880cd307e779fc WatchSource:0}: Error finding container 4fb3530b07cf3d807292acddded7eef0a7e69f32e5efdfd591880cd307e779fc: Status 404 returned error can't find the container with id 4fb3530b07cf3d807292acddded7eef0a7e69f32e5efdfd591880cd307e779fc
Nov 02 13:13:06.663171 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:06.663140    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:13:07.583849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.571861    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" event={"ID":"ffcab597-c6c0-4577-b9fa-a1e20c567b56","Type":"ContainerStarted","Data":"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"}
Nov 02 13:13:07.583849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.571939    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" event={"ID":"ffcab597-c6c0-4577-b9fa-a1e20c567b56","Type":"ContainerStarted","Data":"4fb3530b07cf3d807292acddded7eef0a7e69f32e5efdfd591880cd307e779fc"}
Nov 02 13:13:07.583849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.572132    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:07.583849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.572427    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:07.584708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.584533    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:07.584708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.584578    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" event={"ID":"78f79307-ad18-4efa-bfe4-13d23e590c56","Type":"ContainerStarted","Data":"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"}
Nov 02 13:13:07.584708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.584613    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" event={"ID":"78f79307-ad18-4efa-bfe4-13d23e590c56","Type":"ContainerStarted","Data":"639e1464fdd85f4101e7796685136dfaff7f9c344071537c14d5474203f11f87"}
Nov 02 13:13:07.584903 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.584721    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.606326 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606295    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-plugins-e2e-extmod-a-035dac\""
Nov 02 13:13:07.606487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606362    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"openshift-service-ca.crt\""
Nov 02 13:13:07.606487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606426    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"routing-overlay\""
Nov 02 13:13:07.606487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606296    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"agc-cred-ae68aaa00e1f\""
Nov 02 13:13:07.606667 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606549    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"kube-root-ca.crt\""
Nov 02 13:13:07.606667 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.606582    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-e2e--9bdd4eed-dockercfg-vvx5k\""
Nov 02 13:13:07.610938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.610912    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:07.622709 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.622644    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" podStartSLOduration=2.6226271519999997 podStartE2EDuration="2.622627152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:07.62001239 +0000 UTC m=+2785.269066348" watchObservedRunningTime="2026-11-02 13:13:07.622627152 +0000 UTC m=+2785.271681111"
Nov 02 13:13:07.703662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.703589    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.703896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.703773    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.703975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.703923    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rb2\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.704220 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.704198    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.704278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.704197    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" podStartSLOduration=2.704183712 podStartE2EDuration="2.704183712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:07.703088535 +0000 UTC m=+2785.352142495" watchObservedRunningTime="2026-11-02 13:13:07.704183712 +0000 UTC m=+2785.353237669"
Nov 02 13:13:07.806079 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.806043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.806250 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.806114    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rb2\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.806294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.806262    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.806344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.806327    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.806949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.806918    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.807070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.807045    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.808488 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.808465    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.830388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.830360    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rb2\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:07.925026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:07.924935    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:08.441941 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:13:08.441900    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf4b41b_dd89_4239_a99a_ec303f98bf91.slice/crio-8a2ffd6ff64112ea7789b98d5dda29c7dcac474a810c46663a2af4793544ec04 WatchSource:0}: Error finding container 8a2ffd6ff64112ea7789b98d5dda29c7dcac474a810c46663a2af4793544ec04: Status 404 returned error can't find the container with id 8a2ffd6ff64112ea7789b98d5dda29c7dcac474a810c46663a2af4793544ec04
Nov 02 13:13:08.444060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:08.444024    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:08.699900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:08.687885    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" event={"ID":"7bf4b41b-dd89-4239-a99a-ec303f98bf91","Type":"ContainerStarted","Data":"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"}
Nov 02 13:13:08.699900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:08.687948    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" event={"ID":"7bf4b41b-dd89-4239-a99a-ec303f98bf91","Type":"ContainerStarted","Data":"8a2ffd6ff64112ea7789b98d5dda29c7dcac474a810c46663a2af4793544ec04"}
Nov 02 13:13:08.699900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:08.688579    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:08.763852 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:08.758675    4928 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-035dac/agc-cred-ae68aaa00e1f: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:08.763852 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:08.758703    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:08.763852 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:08.758773    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:09.258754594 +0000 UTC m=+2786.907808547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:09.294124 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:09.291393    4928 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-035dac/agc-cred-ae68aaa00e1f: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:09.294124 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:09.291433    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:09.294124 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:09.291495    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:10.29147628 +0000 UTC m=+2787.940530223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:09.943409 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:09.943362    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:13:10.333059 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:10.333027    4928 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-035dac/agc-cred-ae68aaa00e1f: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:10.333059 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:10.333050    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:10.333303 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:10.333102    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:12.333086045 +0000 UTC m=+2789.982139981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:10.468498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:10.468434    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" podStartSLOduration=3.468414585 podStartE2EDuration="3.468414585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:08.744152331 +0000 UTC m=+2786.393206290" watchObservedRunningTime="2026-11-02 13:13:10.468414585 +0000 UTC m=+2788.117468546"
Nov 02 13:13:10.472416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:10.472386    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:10.754621 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:10.754590    4928 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" secret="" err="secret \"payload-processing-external-model-e2e--9bdd4eed-dockercfg-vvx5k\" not found"
Nov 02 13:13:10.989846 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:10.989710    4928 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-plugins-e2e-extmod-a-035dac: configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:10.989846 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:10.989787    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:11.489767553 +0000 UTC m=+2789.138821495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:11.496528 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:11.496489    4928 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-plugins-e2e-extmod-a-035dac: configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:11.496717 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:11.496612    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:12.496589208 +0000 UTC m=+2790.145643167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:12.357361 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:12.357310    4928 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-035dac/agc-cred-ae68aaa00e1f: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:12.357361 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:12.357346    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:12.358137 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:12.357419    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:16.35739788 +0000 UTC m=+2794.006451837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:12.560776 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:12.560741    4928 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-plugins-e2e-extmod-a-035dac: configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:12.561003 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:12.560849    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:14.560811526 +0000 UTC m=+2792.209865469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:14.594705 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:14.594674    4928 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-plugins-e2e-extmod-a-035dac: configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:14.595153 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:14.594753    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:18.594736641 +0000 UTC m=+2796.243790590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-035dac" not found
Nov 02 13:13:16.427611 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:16.427564    4928 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-035dac/agc-cred-ae68aaa00e1f: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:16.427611 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:16.427598    4928 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll: secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:16.428252 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:16.427648    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials podName:7bf4b41b-dd89-4239-a99a-ec303f98bf91 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:24.427635644 +0000 UTC m=+2802.076689580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91") : secret "agc-cred-ae68aaa00e1f" not found
Nov 02 13:13:16.765625 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:16.765215    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" podUID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" containerName="payload-processing" containerID="cri-o://2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2" gracePeriod=25
Nov 02 13:13:17.065356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.065321    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:17.149793 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.149754    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\") pod \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") "
Nov 02 13:13:17.150048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.149876    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\") pod \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") "
Nov 02 13:13:17.150128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.150108    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\") pod \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") "
Nov 02 13:13:17.150194 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.150173    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\") pod \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\" (UID: \"7bf4b41b-dd89-4239-a99a-ec303f98bf91\") "
Nov 02 13:13:17.151344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.151308    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume" pod "7bf4b41b-dd89-4239-a99a-ec303f98bf91" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:17.151344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.151311    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume" pod "7bf4b41b-dd89-4239-a99a-ec303f98bf91" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:17.154463 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.154433    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2" pod "7bf4b41b-dd89-4239-a99a-ec303f98bf91" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91"). InnerVolumeSpecName "kube-api-access-c5rb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:17.156043 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.156005    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials" pod "7bf4b41b-dd89-4239-a99a-ec303f98bf91" (UID: "7bf4b41b-dd89-4239-a99a-ec303f98bf91"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:17.252215 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.252179    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:17.252215 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.252217    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:17.252442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.252232    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rb2\" (UniqueName: \"kubernetes.io/projected/7bf4b41b-dd89-4239-a99a-ec303f98bf91-kube-api-access-c5rb2\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:17.252442 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.252248    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4b41b-dd89-4239-a99a-ec303f98bf91-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:13:17.932899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.927456    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:13:17.932899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.928723    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" containerName="payload-processing"
Nov 02 13:13:17.932899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.928748    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" containerName="payload-processing"
Nov 02 13:13:17.932899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.928975    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" containerName="payload-processing"
Nov 02 13:13:17.933535 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.933442    4928 generic.go:356] "Generic (PLEG): container finished" podID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" containerID="2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2" exitCode=0
Nov 02 13:13:17.934017 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.933771    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"
Nov 02 13:13:17.937981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.937944    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" event={"ID":"7bf4b41b-dd89-4239-a99a-ec303f98bf91","Type":"ContainerDied","Data":"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"}
Nov 02 13:13:17.938102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.937991    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll" event={"ID":"7bf4b41b-dd89-4239-a99a-ec303f98bf91","Type":"ContainerDied","Data":"8a2ffd6ff64112ea7789b98d5dda29c7dcac474a810c46663a2af4793544ec04"}
Nov 02 13:13:17.938102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.938016    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:13:17.938102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.938049    4928 scope.go:122] "RemoveContainer" containerID="2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"
Nov 02 13:13:17.938296 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.938222    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.960096 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.957435    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f8899-openshift-default-dockercfg-rrzwf\""
Nov 02 13:13:17.966261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.966180    4928 scope.go:122] "RemoveContainer" containerID="2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"
Nov 02 13:13:17.967774 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:17.967713    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2\": container with ID starting with 2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2 not found: ID does not exist" containerID="2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"
Nov 02 13:13:17.967939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.967777    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2"} err="failed to get container status \"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2\": rpc error: code = NotFound desc = could not find container \"2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2\": container with ID starting with 2ef155882a98d1efadc5fafb92ea5f00b89e74c830e1d7c5c892ffbef18398e2 not found: ID does not exist"
Nov 02 13:13:17.977678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.977654    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.977893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.977685    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.977893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.977721    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.977887    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.977931    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.978136    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.978169    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgvf\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.978200    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:17.978322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:17.978262    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.057399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.057366    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:18.066687 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.066630    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5jf5ll"]
Nov 02 13:13:18.079795 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.079756    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.079951    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.079978    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080008    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080035    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080068    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080264    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080295    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgvf\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080324    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080409    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080764 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080734    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.080992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.080961    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.081252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.081231    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.081343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.081221    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.082966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.082945    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.082966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.082955    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.108251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.108224    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.108522 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.108495    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgvf\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.282202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.282159    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:18.655892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.652242    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf4b41b-dd89-4239-a99a-ec303f98bf91" path="/var/lib/kubelet/pods/7bf4b41b-dd89-4239-a99a-ec303f98bf91/volumes"
Nov 02 13:13:18.712294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.710995    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:13:18.712620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:18.712595    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:13:19.237800 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:19.237760    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:13:19.303585 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:19.303537    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:13:19.303927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:19.303897    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:13:19.304122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:19.304098    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:13:20.372983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:20.372943    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerStarted","Data":"6e713a17cfffbe5a27ca13cd2efc0d22b3e842a6b1a3e1b75225657e66985a46"}
Nov 02 13:13:20.394048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:20.394009    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerStarted","Data":"80fb6a8e66262f0c73528c5708a8f39ed430e8f377adc53a8c45db369ad90a20"}
Nov 02 13:13:21.304107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:21.304053    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:21.360540 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:21.360191    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:21.431882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:21.431854    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:21.449765 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:21.449699    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podStartSLOduration=4.449678974 podStartE2EDuration="4.449678974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:20.497656299 +0000 UTC m=+2798.146710257" watchObservedRunningTime="2026-11-02 13:13:21.449678974 +0000 UTC m=+2799.098732933"
Nov 02 13:13:21.462908 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:21.450206    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:13:39.730082 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:39.730034    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:13:47.921673 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:47.921637    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:13:47.927476 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:47.927454    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:47.943620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:47.943592    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6563ed-serving-cert\""
Nov 02 13:13:47.963765 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:47.963734    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:13:48.052969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.052929    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9l9\" (UniqueName: \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.052969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.052975    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.053176 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.053061    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.053219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.053162    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.157698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.157097    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9l9\" (UniqueName: \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.157698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.157163    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.157698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.157227    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.157698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.157305    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.157698 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:48.157677    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-serving-cert: secret "maas-api-e2e-inf-a-6563ed-serving-cert" not found
Nov 02 13:13:48.158178 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:13:48.157752    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls podName:04c637e6-143b-4813-b498-49e8cded4614 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:48.657731067 +0000 UTC m=+2826.306785004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls") pod "maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" (UID: "04c637e6-143b-4813-b498-49e8cded4614") : secret "maas-api-e2e-inf-a-6563ed-serving-cert" not found
Nov 02 13:13:48.176901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.172465    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.176901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.174499    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.183211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.183177    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9l9\" (UniqueName: \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.665915 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.665877    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.668679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.668648    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") pod \"maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:48.845220 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:48.845184    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:49.203062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:49.201531    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:13:49.212637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:49.212611    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:13:49.795905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:49.795853    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" event={"ID":"04c637e6-143b-4813-b498-49e8cded4614","Type":"ContainerStarted","Data":"938ba0540d5be2fe2477de354c238edd3ec0bb530e1e2db6571f762dfc6ef9a8"}
Nov 02 13:13:49.796101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:49.795921    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" event={"ID":"04c637e6-143b-4813-b498-49e8cded4614","Type":"ContainerStarted","Data":"a52617748872f27a4b03175a17c785b623b02f4b52d607c41fccc54162360945"}
Nov 02 13:13:49.796101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:49.795943    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:55.809720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:55.809639    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:13:55.860373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:13:55.860297    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" podStartSLOduration=8.519984108 podStartE2EDuration="8.860279368s" totalImagesPullingTime="340.29526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:49.860140548 +0000 UTC m=+2827.509194531" watchObservedRunningTime="2026-11-02 13:13:55.860279368 +0000 UTC m=+2833.509333327"
Nov 02 13:14:09.736161 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:14:09.736104    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:14:39.760251 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:14:39.760212    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:14:40.410549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.409685    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": read tcp 10.132.0.2:39770->10.132.0.193:15021: read: connection reset by peer" start-of-body=
Nov 02 13:14:40.410549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.409771    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": read tcp 10.132.0.2:39770->10.132.0.193:15021: read: connection reset by peer"
Nov 02 13:14:40.552952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.552799    4928 generic.go:356] "Generic (PLEG): container finished" podID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerID="314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7" exitCode=0
Nov 02 13:14:40.552952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.552893    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerDied","Data":"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"}
Nov 02 13:14:40.553521 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.553488    4928 scope.go:122] "RemoveContainer" containerID="314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"
Nov 02 13:14:40.556063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.556023    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:40.556208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.556108    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:40.556208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:40.556146    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:41.588147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:41.588107    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerStarted","Data":"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"}
Nov 02 13:14:41.909432 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:41.909344    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:14:41.909599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:41.909447    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:41.909599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:41.909494    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:41.997879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:41.992766    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:14:42.012093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.012061    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:14:42.012440 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.012369    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.099566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.099531    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.099852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.099815    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100007 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.099988    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100346    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100604    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100748    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100869    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100911    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4fw\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.100963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.100939    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.101623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.101602    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-361c3a-openshift-default-dockercfg-8nrsz\""
Nov 02 13:14:42.222943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.211810    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.223253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.223218    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.223598 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.223573    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.223819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.223800    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.223987 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.223971    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.232002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.231972    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.232197 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.232177    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4fw\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.239068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.239036    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.249852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.239365    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.249852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.230882    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.250127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.230561    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.250580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.250552    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.276922 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.273521    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.385288 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.308952    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.391928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.254436    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.392164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.378745    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4fw\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.403736 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.403637    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.470587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.467138    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.719373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.719338    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-361c3a-openshift-default-dockercfg-8nrsz\""
Nov 02 13:14:42.724566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.724542    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:42.910216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.909529    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:42.910216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:42.909598    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:43.298423 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:14:43.291772    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53378b8_f9ab_4c5a_83dc_f6bed24b7aab.slice/crio-4e7dc54e0f2329687ac4f39f7b7a21c719f04002a66a932e5178c51aa13c0c55 WatchSource:0}: Error finding container 4e7dc54e0f2329687ac4f39f7b7a21c719f04002a66a932e5178c51aa13c0c55: Status 404 returned error can't find the container with id 4e7dc54e0f2329687ac4f39f7b7a21c719f04002a66a932e5178c51aa13c0c55
Nov 02 13:14:43.298423 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.292883    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:14:43.301926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.301881    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:43.302060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.301973    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:43.302060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.302022    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:43.622653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.622567    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerStarted","Data":"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"}
Nov 02 13:14:43.622653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.622622    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerStarted","Data":"4e7dc54e0f2329687ac4f39f7b7a21c719f04002a66a932e5178c51aa13c0c55"}
Nov 02 13:14:43.719230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.719163    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podStartSLOduration=2.719142005 podStartE2EDuration="2.719142005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:43.713228301 +0000 UTC m=+2881.362282261" watchObservedRunningTime="2026-11-02 13:14:43.719142005 +0000 UTC m=+2881.368195964"
Nov 02 13:14:43.743855 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.732912    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:43.910965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.910279    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:43.910965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:43.910375    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:44.749163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:44.749123    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:44.931881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:44.928666    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:44.931881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:44.928731    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:45.670283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:45.670242    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:45.672578 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:45.672547    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:14:45.909563 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:45.909524    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:45.910020 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:45.909594    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:46.909357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:46.909321    4928 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 02 13:14:46.909545 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:46.909375    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 02 13:14:47.913887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:47.913849    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:14:47.914399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:47.914380    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:14:47.915303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:47.915282    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:14:48.256332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.256215    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:14:48.261759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.261736    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.292362 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.292329    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:14:48.298902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.297908    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:14:48.307910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.306179    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.381729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.359652    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:14:48.381729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.381355    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-361c3a-dockercfg-gqqmv\""
Nov 02 13:14:48.448850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.393216    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-361c3a\""
Nov 02 13:14:48.527523 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.527481    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.527815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.527789    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.535763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.535727    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8ds\" (UniqueName: \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.536122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.536095    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.536368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.536349    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.536574 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.536558    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.542915 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.536710    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.615454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.615414    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.615765 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.615723    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.615936 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.615907    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbp4b\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.616020 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.615998    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.696443 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.696410    4928 generic.go:356] "Generic (PLEG): container finished" podID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerID="6e713a17cfffbe5a27ca13cd2efc0d22b3e842a6b1a3e1b75225657e66985a46" exitCode=0
Nov 02 13:14:48.696620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.696485    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerDied","Data":"6e713a17cfffbe5a27ca13cd2efc0d22b3e842a6b1a3e1b75225657e66985a46"}
Nov 02 13:14:48.697223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.697202    4928 scope.go:122] "RemoveContainer" containerID="6e713a17cfffbe5a27ca13cd2efc0d22b3e842a6b1a3e1b75225657e66985a46"
Nov 02 13:14:48.699033 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.699001    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:48.699158 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.699107    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:48.699226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.699156    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:48.717269 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717246    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp4b\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717312    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717413 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717387    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717452 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717419    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717508 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717451    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8ds\" (UniqueName: \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.717559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717531    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717647 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717627    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.717870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.717851    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718016    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718054    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718080    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.718148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718113    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718294    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718524 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718501    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718561    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.718670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.718639    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.719031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.719007    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.720809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.720788    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.721047 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.721027    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.742770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.742728    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.742957 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.742802    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp4b\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.756669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.756605    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8ds\" (UniqueName: \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\") pod \"payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:48.780982 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.780894    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:48.898872 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:48.898564    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:49.132035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.131998    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:14:49.134251 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:14:49.134219    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398d8f4d_80fe_4e36_b701_2468d193bac8.slice/crio-06b57f43cc89e8293aeb8da243a1856b74303667b374a44c0193a2664fea10ac WatchSource:0}: Error finding container 06b57f43cc89e8293aeb8da243a1856b74303667b374a44c0193a2664fea10ac: Status 404 returned error can't find the container with id 06b57f43cc89e8293aeb8da243a1856b74303667b374a44c0193a2664fea10ac
Nov 02 13:14:49.137725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.137687    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:49.137899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.137760    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:49.137899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.137800    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:14:49.301716 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:14:49.301684    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae833f6_01fa_4f9f_b85e_44730039ae2e.slice/crio-5e8fe30a3718ec5e814659e70201952005a0771c584a5b090765fcb29ea84974 WatchSource:0}: Error finding container 5e8fe30a3718ec5e814659e70201952005a0771c584a5b090765fcb29ea84974: Status 404 returned error can't find the container with id 5e8fe30a3718ec5e814659e70201952005a0771c584a5b090765fcb29ea84974
Nov 02 13:14:49.302686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.302658    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:14:49.722628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.722518    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" event={"ID":"398d8f4d-80fe-4e36-b701-2468d193bac8","Type":"ContainerStarted","Data":"67eb11af8f2ccb6335dfdae848d960ce6ec67e57eefefff8008db526ef9e14cc"}
Nov 02 13:14:49.722628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.722596    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" event={"ID":"398d8f4d-80fe-4e36-b701-2468d193bac8","Type":"ContainerStarted","Data":"06b57f43cc89e8293aeb8da243a1856b74303667b374a44c0193a2664fea10ac"}
Nov 02 13:14:49.725337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.725306    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" event={"ID":"0ae833f6-01fa-4f9f-b85e-44730039ae2e","Type":"ContainerStarted","Data":"e1de749018a221ced99db7083682c13627273b876c559e57779c116b9fee131c"}
Nov 02 13:14:49.725557 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.725540    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" event={"ID":"0ae833f6-01fa-4f9f-b85e-44730039ae2e","Type":"ContainerStarted","Data":"5e8fe30a3718ec5e814659e70201952005a0771c584a5b090765fcb29ea84974"}
Nov 02 13:14:49.726759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.726736    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:14:49.730399 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.730375    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerStarted","Data":"b131a685e8268a2784455494cb3933f973ce6008d46435f5eeec3be1f8d50d4e"}
Nov 02 13:14:49.781511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.781472    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:49.783956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.783911    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podStartSLOduration=1.783896517 podStartE2EDuration="1.783896517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:49.777836817 +0000 UTC m=+2887.426890767" watchObservedRunningTime="2026-11-02 13:14:49.783896517 +0000 UTC m=+2887.432950474"
Nov 02 13:14:49.823619 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:49.823549    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" podStartSLOduration=1.8235293860000001 podStartE2EDuration="1.823529386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:49.820220423 +0000 UTC m=+2887.469274383" watchObservedRunningTime="2026-11-02 13:14:49.823529386 +0000 UTC m=+2887.472583347"
Nov 02 13:14:50.293070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:50.293032    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:14:50.294331 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:50.294304    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 13:14:50.294518 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:50.294495    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 13:14:50.792933 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:50.792530    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:50.792933 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:50.792595    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:51.285917 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:51.285279    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 13:14:51.285917 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:51.285340    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 13:14:51.434851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:51.433934    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:14:51.781883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:51.781846    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:51.782187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:51.782157    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:52.289187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:52.285547    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 13:14:52.289187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:52.285625    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 13:14:52.781337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:52.781297    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:52.782027 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:52.781991    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:53.292773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:53.292236    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 13:14:53.292773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:53.292310    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 13:14:53.783260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:53.783214    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:53.783960 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:53.783924    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:54.288032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:54.287983    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 13:14:54.288355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:54.288324    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 13:14:54.784301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:54.784249    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:54.784926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:54.784878    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:55.324058 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:55.324024    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:14:55.327002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:55.326976    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:14:55.782156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:55.782116    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 13:14:55.782343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:55.782189    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 13:14:56.786881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:56.786847    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:56.787294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:56.787150    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:14:56.787910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:14:56.787892    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:15:00.255439 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.255396    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:00.272447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.271430    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:00.272886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.272807    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.330845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.310175    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:15:00.343048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.343015    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:00.344584 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.343613    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.353378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.353347    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.353560 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.353544    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:00.353923 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.353895    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdfz\" (UniqueName: \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.363616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.363592    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:00.377062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.377038    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:00.380490 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.380466    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.386740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.386710    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:00.386896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.386805    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.393175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.393123    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:00.394751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.394726    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.400026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.399187    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.407117 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.407007    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:00.422721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.422446    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:00.429225 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.429202    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:00.429482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.429463    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.450506 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.450483    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:00.455096 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.455074    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:00.456084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456060    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.456304 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456284    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gwq\" (UniqueName: \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.456430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456415    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4kj\" (UniqueName: \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.456580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456553    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.456652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456603    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.456652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456637    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qp2\" (UniqueName: \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.456767 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456676    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdfz\" (UniqueName: \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.456821 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456783    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.456906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456880    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8trr\" (UniqueName: \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.456978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456913    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz579\" (UniqueName: \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.456978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456949    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.457084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.456976    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjb55\" (UniqueName: \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.457084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.457072    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.457259 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.457192    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.458129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.458107    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.468219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.468156    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:00.482345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.482323    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdfz\" (UniqueName: \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.558370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558272    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.558370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558349    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.558672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558405    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gwq\" (UniqueName: \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.558672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558431    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4kj\" (UniqueName: \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.558672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558488    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.558672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558516    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.558672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558548    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qp2\" (UniqueName: \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.558969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558694    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.558969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558783    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8trr\" (UniqueName: \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.558969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558848    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz579\" (UniqueName: \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.558969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.558884    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjb55\" (UniqueName: \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.559187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.559043    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.559270 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.559241    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.559270 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.559239    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.559685 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.559662    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.559934 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.559909    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.560180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.560161    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.560339 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.560319    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.583481 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.583447    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4kj\" (UniqueName: \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.584263 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.584236    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz579\" (UniqueName: \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.584404 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.584240    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qp2\" (UniqueName: \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.586473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.586448    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjb55\" (UniqueName: \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.586614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.586591    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gwq\" (UniqueName: \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\") pod \"maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:00.592457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.590977    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8trr\" (UniqueName: \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\") pod \"maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.655113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.654986    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:00.692021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.691987    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:00.714234 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.713216    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:00.736263 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.734428    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:00.745224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.744470    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:00.782318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.781882    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:00.790693 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:00.784893    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:01.017868 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:15:01.016962    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63d6edb_92cf_4877_8295_fc326ebd28b5.slice/crio-d245c79c662f8ed8005bdf032058d699145f35efd4f826d2e049e7c45e3e8b3a WatchSource:0}: Error finding container d245c79c662f8ed8005bdf032058d699145f35efd4f826d2e049e7c45e3e8b3a: Status 404 returned error can't find the container with id d245c79c662f8ed8005bdf032058d699145f35efd4f826d2e049e7c45e3e8b3a
Nov 02 13:15:01.021405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.020454    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:01.084319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.084288    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:15:01.092285 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.092247    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.121344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.121312    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:15:01.155775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.155742    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-48d74e-openshift-default-dockercfg-jfvd6\""
Nov 02 13:15:01.230769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.230728    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.270926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.270886    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271114    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5vk\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271187    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271236    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271267    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271385    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271684 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271453    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.271684 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.271580    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.456905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.446528    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.457290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.457254    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.457486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.457468    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.457651 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.457634    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.467396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.457939    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5vk\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.467706 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.467677    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.467876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.467852    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.467987 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.467973    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.468202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.468188    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.477059 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.474552    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.477321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.477169    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.477756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.477726    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.477951 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.477872    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.478169 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.478146    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.558843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.555799    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.558843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.558328    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.588458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.588121    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5vk\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.590797 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.588541    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\") pod \"e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") " pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.592751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.592729    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:01.600526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.600488    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:01.613163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.608521    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:01.616454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.616152    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:01.623712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.623687    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:01.630870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.630844    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:01.709298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.709271    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:01.807283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.806246    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:15:01.914564 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.914462    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerStarted","Data":"a492bfae9e9e195983ad2fdd47522201cd188b04f1a09164466f55267e767b49"}
Nov 02 13:15:01.916133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.916105    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerStarted","Data":"bc5e19794bc358268b62994623aaf57c5abe38f3f852ffb53b800380111e17f8"}
Nov 02 13:15:01.917661 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.917640    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerStarted","Data":"db631706984e0ff28c0abbcb670a4cee0d6c493b3a998e9060fc8591da909a21"}
Nov 02 13:15:01.919374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.919350    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerStarted","Data":"05e6b5158bdf17139e7e20e05d42f773900d208bd0f085ae7a27fdfdffc29b1a"}
Nov 02 13:15:01.920797 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.920775    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerStarted","Data":"f6acc2d9bc9a5c6be2c2136ac00b7c34718a283b09a8d6717e33b86e25801d83"}
Nov 02 13:15:01.922354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.922326    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerStarted","Data":"77e84f90bbb484530e4482b13f88b9e1074b472ea63549f7406cf805bdbc3cb8"}
Nov 02 13:15:01.924527 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.924510    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerStarted","Data":"63eddfb070f2e277f726af4038f529811d07ad505f2d9f105ae4020a196a3873"}
Nov 02 13:15:01.924671 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.924656    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerStarted","Data":"d245c79c662f8ed8005bdf032058d699145f35efd4f826d2e049e7c45e3e8b3a"}
Nov 02 13:15:01.981138 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:01.981082    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" podStartSLOduration=1.981066523 podStartE2EDuration="1.981066523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:01.976322727 +0000 UTC m=+2899.625376686" watchObservedRunningTime="2026-11-02 13:15:01.981066523 +0000 UTC m=+2899.630120481"
Nov 02 13:15:02.264543 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.264404    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:15:02.288872 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:15:02.288822    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf290eea_132b_4d9a_83f0_1edbc5aaca2a.slice/crio-02ba20ba6c1c2c7055da24ce090c2d7ff82fe74f13d2a3d37af12553bc1148ab WatchSource:0}: Error finding container 02ba20ba6c1c2c7055da24ce090c2d7ff82fe74f13d2a3d37af12553bc1148ab: Status 404 returned error can't find the container with id 02ba20ba6c1c2c7055da24ce090c2d7ff82fe74f13d2a3d37af12553bc1148ab
Nov 02 13:15:02.292238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.292203    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:15:02.292319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.292276    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:15:02.292319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.292308    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:15:02.938679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.938643    4928 generic.go:356] "Generic (PLEG): container finished" podID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerID="63eddfb070f2e277f726af4038f529811d07ad505f2d9f105ae4020a196a3873" exitCode=60
Nov 02 13:15:02.938942 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.938729    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerDied","Data":"63eddfb070f2e277f726af4038f529811d07ad505f2d9f105ae4020a196a3873"}
Nov 02 13:15:02.939658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.939154    4928 scope.go:122] "RemoveContainer" containerID="63eddfb070f2e277f726af4038f529811d07ad505f2d9f105ae4020a196a3873"
Nov 02 13:15:02.940856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.940786    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerStarted","Data":"ea78acf62037c9443eab2465faf8686fa36870633654072fa00d63038ed458c7"}
Nov 02 13:15:02.941240 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.941140    4928 scope.go:122] "RemoveContainer" containerID="ea78acf62037c9443eab2465faf8686fa36870633654072fa00d63038ed458c7"
Nov 02 13:15:02.943041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.942844    4928 generic.go:356] "Generic (PLEG): container finished" podID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerID="2b924cd7a2fc39b41cc4043a7b91a210f02d3171a1ed592ad5264c36124177ee" exitCode=60
Nov 02 13:15:02.943041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.942929    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerDied","Data":"2b924cd7a2fc39b41cc4043a7b91a210f02d3171a1ed592ad5264c36124177ee"}
Nov 02 13:15:02.943229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.943210    4928 scope.go:122] "RemoveContainer" containerID="2b924cd7a2fc39b41cc4043a7b91a210f02d3171a1ed592ad5264c36124177ee"
Nov 02 13:15:02.951310 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.950924    4928 generic.go:356] "Generic (PLEG): container finished" podID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerID="b23672768e214517760c5d069b4deca14aa018f67926b3f244bebe5a15371d33" exitCode=60
Nov 02 13:15:02.951310 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.951045    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerDied","Data":"b23672768e214517760c5d069b4deca14aa018f67926b3f244bebe5a15371d33"}
Nov 02 13:15:02.952118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.952097    4928 scope.go:122] "RemoveContainer" containerID="b23672768e214517760c5d069b4deca14aa018f67926b3f244bebe5a15371d33"
Nov 02 13:15:02.953601 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.953565    4928 generic.go:356] "Generic (PLEG): container finished" podID="a4500011-c77a-49ae-845a-24a0cfd73939" containerID="0c9967ede5003c54a38611cbb329812e7f44c716d451bce677424e385f483166" exitCode=60
Nov 02 13:15:02.953740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.953720    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerDied","Data":"0c9967ede5003c54a38611cbb329812e7f44c716d451bce677424e385f483166"}
Nov 02 13:15:02.954372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.954345    4928 scope.go:122] "RemoveContainer" containerID="0c9967ede5003c54a38611cbb329812e7f44c716d451bce677424e385f483166"
Nov 02 13:15:02.956584 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.956561    4928 generic.go:356] "Generic (PLEG): container finished" podID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerID="9f06e1aae2a65dcd1a0e9287bf8cf65ac7bd85bf796aa038125a6333f533bd7a" exitCode=60
Nov 02 13:15:02.956865 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.956845    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerDied","Data":"9f06e1aae2a65dcd1a0e9287bf8cf65ac7bd85bf796aa038125a6333f533bd7a"}
Nov 02 13:15:02.957381 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.957361    4928 scope.go:122] "RemoveContainer" containerID="9f06e1aae2a65dcd1a0e9287bf8cf65ac7bd85bf796aa038125a6333f533bd7a"
Nov 02 13:15:02.959384 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.959358    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerStarted","Data":"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"}
Nov 02 13:15:02.964894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.961896    4928 scope.go:122] "RemoveContainer" containerID="4e2d800ca4a095ab504c276064cff14e5bfe4e74e0e8e37f09ed71c937ea8c37"
Nov 02 13:15:02.965037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.965022    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerStarted","Data":"02ba20ba6c1c2c7055da24ce090c2d7ff82fe74f13d2a3d37af12553bc1148ab"}
Nov 02 13:15:02.965100 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:02.965054    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerStarted","Data":"4e2d800ca4a095ab504c276064cff14e5bfe4e74e0e8e37f09ed71c937ea8c37"}
Nov 02 13:15:03.037224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.037169    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podStartSLOduration=2.037150666 podStartE2EDuration="2.037150666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.03113119 +0000 UTC m=+2900.680185165" watchObservedRunningTime="2026-11-02 13:15:03.037150666 +0000 UTC m=+2900.686204625"
Nov 02 13:15:03.709819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.709784    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:03.716074 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.716044    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:03.975131 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.975095    4928 generic.go:356] "Generic (PLEG): container finished" podID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerID="4e2d800ca4a095ab504c276064cff14e5bfe4e74e0e8e37f09ed71c937ea8c37" exitCode=60
Nov 02 13:15:03.975328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.975216    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerDied","Data":"4e2d800ca4a095ab504c276064cff14e5bfe4e74e0e8e37f09ed71c937ea8c37"}
Nov 02 13:15:03.975328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.975280    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerStarted","Data":"fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"}
Nov 02 13:15:03.977537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.977512    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerStarted","Data":"e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"}
Nov 02 13:15:03.979584 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.979558    4928 generic.go:356] "Generic (PLEG): container finished" podID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerID="ea78acf62037c9443eab2465faf8686fa36870633654072fa00d63038ed458c7" exitCode=60
Nov 02 13:15:03.979683 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.979632    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerDied","Data":"ea78acf62037c9443eab2465faf8686fa36870633654072fa00d63038ed458c7"}
Nov 02 13:15:03.979683 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.979675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerStarted","Data":"824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"}
Nov 02 13:15:03.981605 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.981580    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerStarted","Data":"8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"}
Nov 02 13:15:03.983749 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.983726    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerStarted","Data":"d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"}
Nov 02 13:15:03.985946 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.985924    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerStarted","Data":"9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"}
Nov 02 13:15:03.988147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.988123    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerStarted","Data":"0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"}
Nov 02 13:15:03.988324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.988305    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:03.989309 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:03.989288    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:15:04.034787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.034727    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" podStartSLOduration=4.034709803 podStartE2EDuration="4.034709803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.031475645 +0000 UTC m=+2901.680529600" watchObservedRunningTime="2026-11-02 13:15:04.034709803 +0000 UTC m=+2901.683763762"
Nov 02 13:15:04.153093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.152922    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" podStartSLOduration=4.152901571 podStartE2EDuration="4.152901571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.146666232 +0000 UTC m=+2901.795720190" watchObservedRunningTime="2026-11-02 13:15:04.152901571 +0000 UTC m=+2901.801955530"
Nov 02 13:15:04.202866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.202801    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" podStartSLOduration=4.202784572 podStartE2EDuration="4.202784572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.199485235 +0000 UTC m=+2901.848539192" watchObservedRunningTime="2026-11-02 13:15:04.202784572 +0000 UTC m=+2901.851838530"
Nov 02 13:15:04.249302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.249242    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" podStartSLOduration=4.249227703 podStartE2EDuration="4.249227703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.245658598 +0000 UTC m=+2901.894712551" watchObservedRunningTime="2026-11-02 13:15:04.249227703 +0000 UTC m=+2901.898281690"
Nov 02 13:15:04.299597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.299538    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" podStartSLOduration=4.299525767 podStartE2EDuration="4.299525767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.298133054 +0000 UTC m=+2901.947187219" watchObservedRunningTime="2026-11-02 13:15:04.299525767 +0000 UTC m=+2901.948579725"
Nov 02 13:15:04.353758 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:04.353706    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" podStartSLOduration=4.353688972 podStartE2EDuration="4.353688972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.349980788 +0000 UTC m=+2901.999034748" watchObservedRunningTime="2026-11-02 13:15:04.353688972 +0000 UTC m=+2902.002742929"
Nov 02 13:15:05.001599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.001554    4928 generic.go:356] "Generic (PLEG): container finished" podID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerID="824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e" exitCode=60
Nov 02 13:15:05.002070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.001680    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerDied","Data":"824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"}
Nov 02 13:15:05.002070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.001752    4928 scope.go:122] "RemoveContainer" containerID="ea78acf62037c9443eab2465faf8686fa36870633654072fa00d63038ed458c7"
Nov 02 13:15:05.002070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.002056    4928 scope.go:122] "RemoveContainer" containerID="824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"
Nov 02 13:15:05.002592 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.002554    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz_odh-ai-gateway-infra(893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6"
Nov 02 13:15:05.011681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.011650    4928 generic.go:356] "Generic (PLEG): container finished" podID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerID="8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50" exitCode=60
Nov 02 13:15:05.011943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.011920    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerDied","Data":"8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"}
Nov 02 13:15:05.012596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.012568    4928 scope.go:122] "RemoveContainer" containerID="8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"
Nov 02 13:15:05.013314 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.013280    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r_odh-ai-gateway-infra(d06dbf5f-9c7f-47b0-8bec-c066f5924ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3"
Nov 02 13:15:05.021726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.021695    4928 scope.go:122] "RemoveContainer" containerID="2b924cd7a2fc39b41cc4043a7b91a210f02d3171a1ed592ad5264c36124177ee"
Nov 02 13:15:05.024612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.024252    4928 generic.go:356] "Generic (PLEG): container finished" podID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerID="d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52" exitCode=60
Nov 02 13:15:05.024612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.024347    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerDied","Data":"d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"}
Nov 02 13:15:05.024731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.024711    4928 scope.go:122] "RemoveContainer" containerID="d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"
Nov 02 13:15:05.025142 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.025121    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp_odh-ai-gateway-infra(02df771c-b6a5-48ba-be10-4da1e8fd746a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a"
Nov 02 13:15:05.034330 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.034302    4928 generic.go:356] "Generic (PLEG): container finished" podID="a4500011-c77a-49ae-845a-24a0cfd73939" containerID="9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9" exitCode=60
Nov 02 13:15:05.034454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.034429    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerDied","Data":"9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"}
Nov 02 13:15:05.035109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.035089    4928 scope.go:122] "RemoveContainer" containerID="9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"
Nov 02 13:15:05.035538 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.035514    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c_odh-ai-gateway-infra(a4500011-c77a-49ae-845a-24a0cfd73939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" podUID="a4500011-c77a-49ae-845a-24a0cfd73939"
Nov 02 13:15:05.038015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.037938    4928 generic.go:356] "Generic (PLEG): container finished" podID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerID="0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f" exitCode=60
Nov 02 13:15:05.038092 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.038055    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerDied","Data":"0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"}
Nov 02 13:15:05.038459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.038440    4928 scope.go:122] "RemoveContainer" containerID="0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"
Nov 02 13:15:05.038875 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.038845    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p_odh-ai-gateway-infra(233a44c9-8928-4d12-93e6-b1c3328b58d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3"
Nov 02 13:15:05.040806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.040603    4928 generic.go:356] "Generic (PLEG): container finished" podID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerID="fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22" exitCode=60
Nov 02 13:15:05.040806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.040628    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerDied","Data":"fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"}
Nov 02 13:15:05.041065 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.040967    4928 scope.go:122] "RemoveContainer" containerID="fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"
Nov 02 13:15:05.041322 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.041295    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd_odh-ai-gateway-infra(a6ae7550-efb7-476b-af2f-e9ed51ab49d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2"
Nov 02 13:15:05.043480 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.043453    4928 generic.go:356] "Generic (PLEG): container finished" podID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerID="e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8" exitCode=60
Nov 02 13:15:05.043711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.043670    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerDied","Data":"e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"}
Nov 02 13:15:05.043815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.043778    4928 scope.go:122] "RemoveContainer" containerID="e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"
Nov 02 13:15:05.044130 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:05.044103    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q_odh-ai-gateway-infra(d63d6edb-92cf-4877-8295-fc326ebd28b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5"
Nov 02 13:15:05.047241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.047209    4928 scope.go:122] "RemoveContainer" containerID="b23672768e214517760c5d069b4deca14aa018f67926b3f244bebe5a15371d33"
Nov 02 13:15:05.065931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.065909    4928 scope.go:122] "RemoveContainer" containerID="0c9967ede5003c54a38611cbb329812e7f44c716d451bce677424e385f483166"
Nov 02 13:15:05.083321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.083304    4928 scope.go:122] "RemoveContainer" containerID="9f06e1aae2a65dcd1a0e9287bf8cf65ac7bd85bf796aa038125a6333f533bd7a"
Nov 02 13:15:05.099744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.099718    4928 scope.go:122] "RemoveContainer" containerID="4e2d800ca4a095ab504c276064cff14e5bfe4e74e0e8e37f09ed71c937ea8c37"
Nov 02 13:15:05.114528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:05.114505    4928 scope.go:122] "RemoveContainer" containerID="63eddfb070f2e277f726af4038f529811d07ad505f2d9f105ae4020a196a3873"
Nov 02 13:15:06.098027 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.095684    4928 scope.go:122] "RemoveContainer" containerID="824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"
Nov 02 13:15:06.098027 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.096256    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz_odh-ai-gateway-infra(893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6"
Nov 02 13:15:06.101098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.101057    4928 scope.go:122] "RemoveContainer" containerID="8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"
Nov 02 13:15:06.101535 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.101505    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r_odh-ai-gateway-infra(d06dbf5f-9c7f-47b0-8bec-c066f5924ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3"
Nov 02 13:15:06.105208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.105182    4928 scope.go:122] "RemoveContainer" containerID="d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"
Nov 02 13:15:06.105813 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.105757    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp_odh-ai-gateway-infra(02df771c-b6a5-48ba-be10-4da1e8fd746a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a"
Nov 02 13:15:06.107543 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.107495    4928 scope.go:122] "RemoveContainer" containerID="9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"
Nov 02 13:15:06.109303 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.109210    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c_odh-ai-gateway-infra(a4500011-c77a-49ae-845a-24a0cfd73939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" podUID="a4500011-c77a-49ae-845a-24a0cfd73939"
Nov 02 13:15:06.110179 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.110157    4928 scope.go:122] "RemoveContainer" containerID="0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"
Nov 02 13:15:06.110682 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.110621    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p_odh-ai-gateway-infra(233a44c9-8928-4d12-93e6-b1c3328b58d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3"
Nov 02 13:15:06.112406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.112384    4928 scope.go:122] "RemoveContainer" containerID="fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"
Nov 02 13:15:06.112743 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.112718    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd_odh-ai-gateway-infra(a6ae7550-efb7-476b-af2f-e9ed51ab49d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2"
Nov 02 13:15:06.114456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:06.114436    4928 scope.go:122] "RemoveContainer" containerID="e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"
Nov 02 13:15:06.114725 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:06.114707    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q_odh-ai-gateway-infra(d63d6edb-92cf-4877-8295-fc326ebd28b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5"
Nov 02 13:15:09.731982 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:09.731940    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:15:15.490375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:15.490346    4928 scope.go:122] "RemoveContainer" containerID="e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"
Nov 02 13:15:15.490868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:15.490442    4928 scope.go:122] "RemoveContainer" containerID="fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"
Nov 02 13:15:16.213710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:16.213676    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerStarted","Data":"daf78029ac23b410b7644615dd54997c7eaafd0dc2a6dd2aba5659455cd553d5"}
Nov 02 13:15:16.216070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:16.216041    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerStarted","Data":"fc7a55a1313ee13fbbda5ed12e663554ce9ab51b41a4fbdc5bb476d85072a3df"}
Nov 02 13:15:16.490909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:16.490878    4928 scope.go:122] "RemoveContainer" containerID="0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"
Nov 02 13:15:16.491322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:16.491032    4928 scope.go:122] "RemoveContainer" containerID="d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"
Nov 02 13:15:17.239745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.239701    4928 generic.go:356] "Generic (PLEG): container finished" podID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerID="fc7a55a1313ee13fbbda5ed12e663554ce9ab51b41a4fbdc5bb476d85072a3df" exitCode=60
Nov 02 13:15:17.240206 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.240149    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerDied","Data":"fc7a55a1313ee13fbbda5ed12e663554ce9ab51b41a4fbdc5bb476d85072a3df"}
Nov 02 13:15:17.240305 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.240271    4928 scope.go:122] "RemoveContainer" containerID="e123781f0ff54cdbb41987af9af85c84c95b85eac6dbab1c70cb134dc43987c8"
Nov 02 13:15:17.241513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.241487    4928 scope.go:122] "RemoveContainer" containerID="fc7a55a1313ee13fbbda5ed12e663554ce9ab51b41a4fbdc5bb476d85072a3df"
Nov 02 13:15:17.243727 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:17.243694    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q_odh-ai-gateway-infra(d63d6edb-92cf-4877-8295-fc326ebd28b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5"
Nov 02 13:15:17.244437 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.244406    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerStarted","Data":"cecebc8b6dda53a8c1a24425d8f605395a39262b701bf99eeb2e2918dd4ee76f"}
Nov 02 13:15:17.252078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.252044    4928 generic.go:356] "Generic (PLEG): container finished" podID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerID="daf78029ac23b410b7644615dd54997c7eaafd0dc2a6dd2aba5659455cd553d5" exitCode=60
Nov 02 13:15:17.252189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.252118    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerDied","Data":"daf78029ac23b410b7644615dd54997c7eaafd0dc2a6dd2aba5659455cd553d5"}
Nov 02 13:15:17.252600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.252578    4928 scope.go:122] "RemoveContainer" containerID="daf78029ac23b410b7644615dd54997c7eaafd0dc2a6dd2aba5659455cd553d5"
Nov 02 13:15:17.253095 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:17.253067    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd_odh-ai-gateway-infra(a6ae7550-efb7-476b-af2f-e9ed51ab49d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2"
Nov 02 13:15:17.262156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.262134    4928 scope.go:122] "RemoveContainer" containerID="fcb48e733dc7d784dd3dd862ca327a85cec6c955b771845c8c38a30bc089fd22"
Nov 02 13:15:17.280210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.280187    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:17.315874 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.315738    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:17.490445 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.490413    4928 scope.go:122] "RemoveContainer" containerID="824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"
Nov 02 13:15:17.490624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:17.490556    4928 scope.go:122] "RemoveContainer" containerID="8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"
Nov 02 13:15:18.270358 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.270326    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerStarted","Data":"62a463221347bbc0292f268551aff05c70ec3f5a54d572ace6fb396373077356"}
Nov 02 13:15:18.272892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.272861    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerStarted","Data":"ac4dbfb0778c8608a217b3328461e30dda8667f9d85dd6997270e1e0a43c6824"}
Nov 02 13:15:18.276736 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.276705    4928 generic.go:356] "Generic (PLEG): container finished" podID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerID="a21099fdda55eeee351ca159a54d389caa28e2f0ede2f894b536b09c3ab876e9" exitCode=60
Nov 02 13:15:18.276860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.276774    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerDied","Data":"a21099fdda55eeee351ca159a54d389caa28e2f0ede2f894b536b09c3ab876e9"}
Nov 02 13:15:18.276931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.276876    4928 scope.go:122] "RemoveContainer" containerID="d4a6d55bd6014f7dc7c538353f6325b0d5cf9525ddf718de2493b3aa99c71c52"
Nov 02 13:15:18.277215 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.277201    4928 scope.go:122] "RemoveContainer" containerID="a21099fdda55eeee351ca159a54d389caa28e2f0ede2f894b536b09c3ab876e9"
Nov 02 13:15:18.277606 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:18.277581    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp_odh-ai-gateway-infra(02df771c-b6a5-48ba-be10-4da1e8fd746a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a"
Nov 02 13:15:18.279062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.279036    4928 generic.go:356] "Generic (PLEG): container finished" podID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerID="cecebc8b6dda53a8c1a24425d8f605395a39262b701bf99eeb2e2918dd4ee76f" exitCode=60
Nov 02 13:15:18.279182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.279112    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerDied","Data":"cecebc8b6dda53a8c1a24425d8f605395a39262b701bf99eeb2e2918dd4ee76f"}
Nov 02 13:15:18.279494 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.279477    4928 scope.go:122] "RemoveContainer" containerID="cecebc8b6dda53a8c1a24425d8f605395a39262b701bf99eeb2e2918dd4ee76f"
Nov 02 13:15:18.279810 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:18.279787    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p_odh-ai-gateway-infra(233a44c9-8928-4d12-93e6-b1c3328b58d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3"
Nov 02 13:15:18.296144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.296123    4928 scope.go:122] "RemoveContainer" containerID="0193059d28356c532a7ea524b9ab5301d8edcc8967495f11689bd5f2827a309f"
Nov 02 13:15:18.394450 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.394418    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:18.491315 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.491283    4928 scope.go:122] "RemoveContainer" containerID="9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"
Nov 02 13:15:18.518682 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.518590    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:18.570130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.570101    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:18.598995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.598959    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\" (UniqueName: \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\") pod \"d63d6edb-92cf-4877-8295-fc326ebd28b5\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") "
Nov 02 13:15:18.599205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.599019    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\") pod \"d63d6edb-92cf-4877-8295-fc326ebd28b5\" (UID: \"d63d6edb-92cf-4877-8295-fc326ebd28b5\") "
Nov 02 13:15:18.601704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.601671    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca" pod "d63d6edb-92cf-4877-8295-fc326ebd28b5" (UID: "d63d6edb-92cf-4877-8295-fc326ebd28b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:18.602629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.602598    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz" pod "d63d6edb-92cf-4877-8295-fc326ebd28b5" (UID: "d63d6edb-92cf-4877-8295-fc326ebd28b5"). InnerVolumeSpecName "kube-api-access-nhdfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:18.700628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.700594    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\") pod \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") "
Nov 02 13:15:18.700789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.700753    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\" (UniqueName: \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\") pod \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\" (UID: \"a6ae7550-efb7-476b-af2f-e9ed51ab49d2\") "
Nov 02 13:15:18.701067 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.701037    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca" pod "a6ae7550-efb7-476b-af2f-e9ed51ab49d2" (UID: "a6ae7550-efb7-476b-af2f-e9ed51ab49d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:18.701657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.701628    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.701783 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.701659    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdfz\" (UniqueName: \"kubernetes.io/projected/d63d6edb-92cf-4877-8295-fc326ebd28b5-kube-api-access-nhdfz\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.701783 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.701675    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63d6edb-92cf-4877-8295-fc326ebd28b5-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.728479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.728051    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55" pod "a6ae7550-efb7-476b-af2f-e9ed51ab49d2" (UID: "a6ae7550-efb7-476b-af2f-e9ed51ab49d2"). InnerVolumeSpecName "kube-api-access-bjb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:18.803277 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:18.803202    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjb55\" (UniqueName: \"kubernetes.io/projected/a6ae7550-efb7-476b-af2f-e9ed51ab49d2-kube-api-access-bjb55\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.292685 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.292648    4928 generic.go:356] "Generic (PLEG): container finished" podID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerID="62a463221347bbc0292f268551aff05c70ec3f5a54d572ace6fb396373077356" exitCode=60
Nov 02 13:15:19.293262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.292729    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerDied","Data":"62a463221347bbc0292f268551aff05c70ec3f5a54d572ace6fb396373077356"}
Nov 02 13:15:19.293262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.292835    4928 scope.go:122] "RemoveContainer" containerID="824aeae52334c056856921101c7fe27f739c721754a441eafdb368482414013e"
Nov 02 13:15:19.293387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.293348    4928 scope.go:122] "RemoveContainer" containerID="62a463221347bbc0292f268551aff05c70ec3f5a54d572ace6fb396373077356"
Nov 02 13:15:19.296861 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:19.293816    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz_odh-ai-gateway-infra(893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6"
Nov 02 13:15:19.300360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.300338    4928 generic.go:356] "Generic (PLEG): container finished" podID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerID="ac4dbfb0778c8608a217b3328461e30dda8667f9d85dd6997270e1e0a43c6824" exitCode=60
Nov 02 13:15:19.300489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.300415    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerDied","Data":"ac4dbfb0778c8608a217b3328461e30dda8667f9d85dd6997270e1e0a43c6824"}
Nov 02 13:15:19.300935 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.300915    4928 scope.go:122] "RemoveContainer" containerID="ac4dbfb0778c8608a217b3328461e30dda8667f9d85dd6997270e1e0a43c6824"
Nov 02 13:15:19.301366 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:19.301345    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r_odh-ai-gateway-infra(d06dbf5f-9c7f-47b0-8bec-c066f5924ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3"
Nov 02 13:15:19.305018 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.304990    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerStarted","Data":"eb71e37a097e46cbbff9a15419e6f022d98891f332b07e2d081b91302d686424"}
Nov 02 13:15:19.308509 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.308487    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"
Nov 02 13:15:19.308612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.308534    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd" event={"ID":"a6ae7550-efb7-476b-af2f-e9ed51ab49d2","Type":"ContainerDied","Data":"77e84f90bbb484530e4482b13f88b9e1074b472ea63549f7406cf805bdbc3cb8"}
Nov 02 13:15:19.310446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.310425    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"
Nov 02 13:15:19.310538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.310466    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q" event={"ID":"d63d6edb-92cf-4877-8295-fc326ebd28b5","Type":"ContainerDied","Data":"d245c79c662f8ed8005bdf032058d699145f35efd4f826d2e049e7c45e3e8b3a"}
Nov 02 13:15:19.314643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.314621    4928 scope.go:122] "RemoveContainer" containerID="8bf0cdbc74cc87f727e0d5700a8bf7edaa453a10c8c62de244b2f3305a510e50"
Nov 02 13:15:19.332236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.332217    4928 scope.go:122] "RemoveContainer" containerID="daf78029ac23b410b7644615dd54997c7eaafd0dc2a6dd2aba5659455cd553d5"
Nov 02 13:15:19.347178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.347152    4928 scope.go:122] "RemoveContainer" containerID="fc7a55a1313ee13fbbda5ed12e663554ce9ab51b41a4fbdc5bb476d85072a3df"
Nov 02 13:15:19.393476 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.393425    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:19.399698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.399667    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893755-jh5gd"]
Nov 02 13:15:19.418675 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.418644    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:19.518572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.518540    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:19.524447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.524418    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:19.551380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.551293    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:19.584933 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.584896    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:19.591100 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.591057    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893755-pt66q"]
Nov 02 13:15:19.622997 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.621764    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\" (UniqueName: \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\") pod \"233a44c9-8928-4d12-93e6-b1c3328b58d3\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") "
Nov 02 13:15:19.622997 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.622069    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\") pod \"233a44c9-8928-4d12-93e6-b1c3328b58d3\" (UID: \"233a44c9-8928-4d12-93e6-b1c3328b58d3\") "
Nov 02 13:15:19.623610 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.623538    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca" pod "233a44c9-8928-4d12-93e6-b1c3328b58d3" (UID: "233a44c9-8928-4d12-93e6-b1c3328b58d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:19.634733 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.634704    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2" pod "233a44c9-8928-4d12-93e6-b1c3328b58d3" (UID: "233a44c9-8928-4d12-93e6-b1c3328b58d3"). InnerVolumeSpecName "kube-api-access-x4qp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:19.725161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.725057    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qp2\" (UniqueName: \"kubernetes.io/projected/233a44c9-8928-4d12-93e6-b1c3328b58d3-kube-api-access-x4qp2\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.725161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.725087    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/233a44c9-8928-4d12-93e6-b1c3328b58d3-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.761212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.761179    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:19.927193 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.927111    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\") pod \"02df771c-b6a5-48ba-be10-4da1e8fd746a\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") "
Nov 02 13:15:19.927353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.927256    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\" (UniqueName: \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\") pod \"02df771c-b6a5-48ba-be10-4da1e8fd746a\" (UID: \"02df771c-b6a5-48ba-be10-4da1e8fd746a\") "
Nov 02 13:15:19.927505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.927480    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca" pod "02df771c-b6a5-48ba-be10-4da1e8fd746a" (UID: "02df771c-b6a5-48ba-be10-4da1e8fd746a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:19.928007 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.927974    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02df771c-b6a5-48ba-be10-4da1e8fd746a-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.929526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:19.929498    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq" pod "02df771c-b6a5-48ba-be10-4da1e8fd746a" (UID: "02df771c-b6a5-48ba-be10-4da1e8fd746a"). InnerVolumeSpecName "kube-api-access-x6gwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.031493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.031447    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6gwq\" (UniqueName: \"kubernetes.io/projected/02df771c-b6a5-48ba-be10-4da1e8fd746a-kube-api-access-x6gwq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.336261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.336226    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp" event={"ID":"02df771c-b6a5-48ba-be10-4da1e8fd746a","Type":"ContainerDied","Data":"db631706984e0ff28c0abbcb670a4cee0d6c493b3a998e9060fc8591da909a21"}
Nov 02 13:15:20.336657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.336301    4928 scope.go:122] "RemoveContainer" containerID="a21099fdda55eeee351ca159a54d389caa28e2f0ede2f894b536b09c3ab876e9"
Nov 02 13:15:20.336657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.336413    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"
Nov 02 13:15:20.340274 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.340242    4928 generic.go:356] "Generic (PLEG): container finished" podID="a4500011-c77a-49ae-845a-24a0cfd73939" containerID="eb71e37a097e46cbbff9a15419e6f022d98891f332b07e2d081b91302d686424" exitCode=60
Nov 02 13:15:20.340414 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.340368    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerDied","Data":"eb71e37a097e46cbbff9a15419e6f022d98891f332b07e2d081b91302d686424"}
Nov 02 13:15:20.340874 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.340802    4928 scope.go:122] "RemoveContainer" containerID="eb71e37a097e46cbbff9a15419e6f022d98891f332b07e2d081b91302d686424"
Nov 02 13:15:20.341227 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:20.341199    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c_odh-ai-gateway-infra(a4500011-c77a-49ae-845a-24a0cfd73939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" podUID="a4500011-c77a-49ae-845a-24a0cfd73939"
Nov 02 13:15:20.349883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.345570    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"
Nov 02 13:15:20.349883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.345581    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p" event={"ID":"233a44c9-8928-4d12-93e6-b1c3328b58d3","Type":"ContainerDied","Data":"f6acc2d9bc9a5c6be2c2136ac00b7c34718a283b09a8d6717e33b86e25801d83"}
Nov 02 13:15:20.358756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.358727    4928 scope.go:122] "RemoveContainer" containerID="9a63098aaa8a238de0bb8add373b3c5f5192e734e4e84c1ea8019ba4ff2b76a9"
Nov 02 13:15:20.378884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.378796    4928 scope.go:122] "RemoveContainer" containerID="cecebc8b6dda53a8c1a24425d8f605395a39262b701bf99eeb2e2918dd4ee76f"
Nov 02 13:15:20.508626 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.508590    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" path="/var/lib/kubelet/pods/a6ae7550-efb7-476b-af2f-e9ed51ab49d2/volumes"
Nov 02 13:15:20.509235 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.509209    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" path="/var/lib/kubelet/pods/d63d6edb-92cf-4877-8295-fc326ebd28b5/volumes"
Nov 02 13:15:20.539810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.539777    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:20.544015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.543988    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bbb7d0-29893755-kqcgp"]
Nov 02 13:15:20.557038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.557013    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:20.563812 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.563784    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:20.571050 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.571023    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:20.591995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.591917    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:20.599729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.599703    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893755-hll8p"]
Nov 02 13:15:20.750647 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.750606    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\") pod \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") "
Nov 02 13:15:20.750879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.750661    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\" (UniqueName: \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\") pod \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\" (UID: \"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3\") "
Nov 02 13:15:20.750879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.750772    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\") pod \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") "
Nov 02 13:15:20.751025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.750946    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\" (UniqueName: \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\") pod \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\" (UID: \"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6\") "
Nov 02 13:15:20.751125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.751080    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca" pod "d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" (UID: "d06dbf5f-9c7f-47b0-8bec-c066f5924ab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.751189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.751164    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca" pod "893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" (UID: "893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.751711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.751689    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.751876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.751713    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.753372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.753345    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579" pod "893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" (UID: "893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6"). InnerVolumeSpecName "kube-api-access-hz579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.753497 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.753441    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj" pod "d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" (UID: "d06dbf5f-9c7f-47b0-8bec-c066f5924ab3"). InnerVolumeSpecName "kube-api-access-lb4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.853780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.853695    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4kj\" (UniqueName: \"kubernetes.io/projected/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3-kube-api-access-lb4kj\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.853780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:20.853732    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz579\" (UniqueName: \"kubernetes.io/projected/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6-kube-api-access-hz579\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:21.361805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.361772    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"
Nov 02 13:15:21.362282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.361802    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz" event={"ID":"893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6","Type":"ContainerDied","Data":"a492bfae9e9e195983ad2fdd47522201cd188b04f1a09164466f55267e767b49"}
Nov 02 13:15:21.362282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.361890    4928 scope.go:122] "RemoveContainer" containerID="62a463221347bbc0292f268551aff05c70ec3f5a54d572ace6fb396373077356"
Nov 02 13:15:21.363583 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.363563    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"
Nov 02 13:15:21.363739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.363584    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r" event={"ID":"d06dbf5f-9c7f-47b0-8bec-c066f5924ab3","Type":"ContainerDied","Data":"bc5e19794bc358268b62994623aaf57c5abe38f3f852ffb53b800380111e17f8"}
Nov 02 13:15:21.378331 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.378302    4928 scope.go:122] "RemoveContainer" containerID="ac4dbfb0778c8608a217b3328461e30dda8667f9d85dd6997270e1e0a43c6824"
Nov 02 13:15:21.421490 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.421455    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:21.432706 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.432675    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893755-59s4r"]
Nov 02 13:15:21.518188 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.518154    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:21.527129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.527101    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893755-ttdmz"]
Nov 02 13:15:21.575294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.575261    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:21.769271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.769233    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\" (UniqueName: \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\") pod \"a4500011-c77a-49ae-845a-24a0cfd73939\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") "
Nov 02 13:15:21.769484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.769376    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\") pod \"a4500011-c77a-49ae-845a-24a0cfd73939\" (UID: \"a4500011-c77a-49ae-845a-24a0cfd73939\") "
Nov 02 13:15:21.769729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.769700    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca" pod "a4500011-c77a-49ae-845a-24a0cfd73939" (UID: "a4500011-c77a-49ae-845a-24a0cfd73939"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:21.770239 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.770214    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4500011-c77a-49ae-845a-24a0cfd73939-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:21.771709 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.771680    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr" pod "a4500011-c77a-49ae-845a-24a0cfd73939" (UID: "a4500011-c77a-49ae-845a-24a0cfd73939"). InnerVolumeSpecName "kube-api-access-m8trr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:21.928556 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:21.928477    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8trr\" (UniqueName: \"kubernetes.io/projected/a4500011-c77a-49ae-845a-24a0cfd73939-kube-api-access-m8trr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:15:22.384132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.384103    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"
Nov 02 13:15:22.384588 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.384102    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c" event={"ID":"a4500011-c77a-49ae-845a-24a0cfd73939","Type":"ContainerDied","Data":"05e6b5158bdf17139e7e20e05d42f773900d208bd0f085ae7a27fdfdffc29b1a"}
Nov 02 13:15:22.384588 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.384258    4928 scope.go:122] "RemoveContainer" containerID="eb71e37a097e46cbbff9a15419e6f022d98891f332b07e2d081b91302d686424"
Nov 02 13:15:22.443861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.443810    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:22.449393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.449361    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f8899-29893755-ftw2c"]
Nov 02 13:15:22.500654 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.500623    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" path="/var/lib/kubelet/pods/02df771c-b6a5-48ba-be10-4da1e8fd746a/volumes"
Nov 02 13:15:22.501189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.501172    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" path="/var/lib/kubelet/pods/233a44c9-8928-4d12-93e6-b1c3328b58d3/volumes"
Nov 02 13:15:22.501623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.501608    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" path="/var/lib/kubelet/pods/893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6/volumes"
Nov 02 13:15:22.502275 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.502258    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" path="/var/lib/kubelet/pods/a4500011-c77a-49ae-845a-24a0cfd73939/volumes"
Nov 02 13:15:22.502696 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:22.502682    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" path="/var/lib/kubelet/pods/d06dbf5f-9c7f-47b0-8bec-c066f5924ab3/volumes"
Nov 02 13:15:35.379968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.379876    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:15:35.382429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382393    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.382429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382426    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382457    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382465    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382482    4928 cpu_manager.go:414] "containerMap: removing container" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382490    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382505    4928 cpu_manager.go:414] "containerMap: removing container" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382512    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382529    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382537    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382555    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382562    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382593    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382601    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382616    4928 cpu_manager.go:414] "containerMap: removing container" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.382635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382623    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382652    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382659    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382668    4928 cpu_manager.go:414] "containerMap: removing container" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382675    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382697    4928 cpu_manager.go:414] "containerMap: removing container" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382703    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382723    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382730    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382747    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382754    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382783    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382792    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382807    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382813    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382870    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382877    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382897    4928 cpu_manager.go:414] "containerMap: removing container" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.382909    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383268    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383285    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383306    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383315    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383325    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383342    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383356    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383365    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.383370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383375    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383391    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383410    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383425    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383444    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383464    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:15:35.384566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.383473    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:15:35.390771 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.390739    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.407353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.407326    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:15:35.412186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.412158    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-361c3a-serving-cert\""
Nov 02 13:15:35.493284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.493248    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5m42\" (UniqueName: \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.493434 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.493292    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.493434 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.493313    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.493525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.493486    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.598662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.598263    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.598662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.598318    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.598662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.598499    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.599003 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.598906    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5m42\" (UniqueName: \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.601295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.600320    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.601295 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:35.600436    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-serving-cert: secret "maas-api-e2e-inf-b-361c3a-serving-cert" not found
Nov 02 13:15:35.601295 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:35.600499    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls podName:d95d9c1b-9742-4173-add0-df61d21600a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:15:36.100478317 +0000 UTC m=+2933.749532267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls") pod "maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" (UID: "d95d9c1b-9742-4173-add0-df61d21600a9") : secret "maas-api-e2e-inf-b-361c3a-serving-cert" not found
Nov 02 13:15:35.611483 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.610533    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:35.653907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:35.651791    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5m42\" (UniqueName: \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:36.107014 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:36.106976    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:36.109478 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:36.109451    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:36.308780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:36.308748    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:36.630686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:36.630657    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:15:36.632442 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:15:36.632417    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95d9c1b_9742_4173_add0_df61d21600a9.slice/crio-d4592679b8af85ade8da97d9a289dfc369720dc9de6e973981efbdd2806dbb2d WatchSource:0}: Error finding container d4592679b8af85ade8da97d9a289dfc369720dc9de6e973981efbdd2806dbb2d: Status 404 returned error can't find the container with id d4592679b8af85ade8da97d9a289dfc369720dc9de6e973981efbdd2806dbb2d
Nov 02 13:15:37.601728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:37.601690    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" event={"ID":"d95d9c1b-9742-4173-add0-df61d21600a9","Type":"ContainerStarted","Data":"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"}
Nov 02 13:15:37.601962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:37.601744    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" event={"ID":"d95d9c1b-9742-4173-add0-df61d21600a9","Type":"ContainerStarted","Data":"d4592679b8af85ade8da97d9a289dfc369720dc9de6e973981efbdd2806dbb2d"}
Nov 02 13:15:37.601962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:37.601785    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:15:37.650134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:37.650062    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" podStartSLOduration=2.305404551 podStartE2EDuration="2.650043925s" totalImagesPullingTime="344.639374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:37.646332851 +0000 UTC m=+2935.295386813" watchObservedRunningTime="2026-11-02 13:15:37.650043925 +0000 UTC m=+2935.299097883"
Nov 02 13:15:39.730926 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:15:39.730881    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:15:43.616684 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:15:43.616655    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:16:09.759003 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:16:09.758953    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:16:15.147935 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.147892    4928 generic.go:356] "Generic (PLEG): container finished" podID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerID="d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc" exitCode=0
Nov 02 13:16:15.148427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.147959    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerDied","Data":"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"}
Nov 02 13:16:15.148482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.148466    4928 scope.go:122] "RemoveContainer" containerID="d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"
Nov 02 13:16:15.150687 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.150654    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:15.150776 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.150732    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:15.150815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.150774    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:15.669657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:15.669626    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:16:16.167313 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:16.167279    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerStarted","Data":"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"}
Nov 02 13:16:16.740851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:16.732984    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:16:16.740851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:16.735082    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 13:16:16.740851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:16.735158    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 13:16:17.731852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:17.729289    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 13:16:17.731852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:17.729367    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 13:16:18.725247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:18.725208    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 13:16:18.725608 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:18.725573    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 13:16:19.732287 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:19.732247    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 13:16:19.732965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:19.732930    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 13:16:20.728779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:20.728298    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 13:16:20.728779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:20.728374    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 13:16:21.732088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:21.732054    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:16:21.732512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:21.732484    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:16:21.733180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:21.733159    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:16:39.731459 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:16:39.731412    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:16:43.165862 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.165715    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:16:43.180768 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.180740    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:16:43.679528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.679488    4928 generic.go:356] "Generic (PLEG): container finished" podID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerID="901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b" exitCode=0
Nov 02 13:16:43.679713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.679533    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerDied","Data":"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"}
Nov 02 13:16:43.679995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.679972    4928 scope.go:122] "RemoveContainer" containerID="901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"
Nov 02 13:16:43.682022 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.681988    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:43.682137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.682057    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:43.682137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:43.682090    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:16:44.690625 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:44.690583    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerStarted","Data":"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"}
Nov 02 13:16:44.710147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:44.710117    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:16:44.711140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:44.711111    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:44.711279 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:44.711163    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:45.712851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:45.710688    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:45.712851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:45.710752    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:46.709695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:46.709662    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:46.709914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:46.709716    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:47.710366 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:47.710322    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:47.710981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:47.710391    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:48.710101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:48.710065    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:48.710301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:48.710118    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:48.988607 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:48.988568    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:16:49.710190 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:49.710148    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:16:49.710398 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:49.710227    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 13:16:50.901780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:50.901742    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:16:50.903356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:16:50.903332    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:17:06.101902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.099462    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:17:06.132333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.132299    4928 cpu_manager.go:414] "containerMap: removing container" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:17:06.132638 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.132618    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:17:06.132799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.132786    4928 cpu_manager.go:414] "containerMap: removing container" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:17:06.132929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.132914    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:17:06.133164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133148    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:17:06.133267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133254    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06dbf5f-9c7f-47b0-8bec-c066f5924ab3" containerName="cleanup"
Nov 02 13:17:06.133375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133364    4928 cpu_manager.go:414] "containerMap: removing container" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:17:06.133453 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133443    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:17:06.133811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133793    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4500011-c77a-49ae-845a-24a0cfd73939" containerName="cleanup"
Nov 02 13:17:06.133961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.133947    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="02df771c-b6a5-48ba-be10-4da1e8fd746a" containerName="cleanup"
Nov 02 13:17:06.134056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.134045    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="893478f8-3bd8-45f4-9ab7-e2d6baa2f7f6" containerName="cleanup"
Nov 02 13:17:06.134145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.134134    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="233a44c9-8928-4d12-93e6-b1c3328b58d3" containerName="cleanup"
Nov 02 13:17:06.134243 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.134232    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae7550-efb7-476b-af2f-e9ed51ab49d2" containerName="cleanup"
Nov 02 13:17:06.134322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.134312    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63d6edb-92cf-4877-8295-fc326ebd28b5" containerName="cleanup"
Nov 02 13:17:06.151441 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.151415    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:17:06.152513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.152445    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.178805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.178774    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.187492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.187458    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.187793 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.187768    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.188047 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.188026    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.189130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.189108    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.189344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.189323    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.189688 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.189667    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrpx\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.191511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.191488    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.388092 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.387891    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.388696 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.186399    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e8a402-openshift-default-dockercfg-zrpdb\""
Nov 02 13:17:06.490059 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490019    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490118    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490140    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490180    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490262    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490284    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490402    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrpx\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490419    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490435    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490635    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.490907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.490883    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.491106 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.491012    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.491284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.491263    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.491371 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.491275    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.493176 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.493149    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.493280 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.493242    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.515040 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.515015    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrpx\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.518489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.518458    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\") pod \"e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") " pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:06.689573 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:06.689478    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:07.037681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:07.037649    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:17:07.039105 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:17:07.039073    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7344f928_7931_47b9_b15a_58fc25d6e878.slice/crio-2af26a364777926370d77c655752e5bba41ce903a5494e656327313443573333 WatchSource:0}: Error finding container 2af26a364777926370d77c655752e5bba41ce903a5494e656327313443573333: Status 404 returned error can't find the container with id 2af26a364777926370d77c655752e5bba41ce903a5494e656327313443573333
Nov 02 13:17:07.042212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:07.042173    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:07.042291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:07.042247    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:07.042291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:07.042277    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:08.052841 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:08.052782    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerStarted","Data":"151c63225a24b8706d6435cc186c4f9960566a28f60ff1120f55c56a2f7156bc"}
Nov 02 13:17:08.053380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:08.052889    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerStarted","Data":"2af26a364777926370d77c655752e5bba41ce903a5494e656327313443573333"}
Nov 02 13:17:08.150635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:08.150579    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podStartSLOduration=2.150565303 podStartE2EDuration="2.150565303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:08.137280119 +0000 UTC m=+3025.786334080" watchObservedRunningTime="2026-11-02 13:17:08.150565303 +0000 UTC m=+3025.799619261"
Nov 02 13:17:08.690637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:08.690606    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:08.697555 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:08.697517    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:09.061098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:09.061058    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:09.062303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:09.062277    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:17:09.733037 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:17:09.732992    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:17:29.480794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:29.480753    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:17:30.276621 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.276588    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:17:30.282811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.282786    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.296962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.296936    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:17:30.301490 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.301470    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.304653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.304635    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6563ed-openshift-default-dockercfg-dh4qv\""
Nov 02 13:17:30.313412 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.313385    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:17:30.333192 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333152    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.333486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333460    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvckx\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.333585 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333499    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.333694 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333676    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.333755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333739    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.334298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333781    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.334298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.333964    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.334298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.334002    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.334298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.334040    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.341303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.341275    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:17:30.435483 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435444    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.435660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435528    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.435660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435637    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvckx\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.435752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435675    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.435752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435715    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.435752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435744    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.435911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.435973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.435948    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.436068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436045    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.436136 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436110    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.436198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436157    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436233    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436316 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436275    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436327    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtpsp\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.436525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436496    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436595 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436517    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.436641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436592    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436636    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436665    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436776 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436705    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.436924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436899    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.437011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.436960    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.437156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.437136    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.438741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.438710    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.439417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.439391    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.460770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.460728    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvckx\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.460924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.460897    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.538226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538122    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.538677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538270    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.538677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538300    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.538677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538375    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.538677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538426    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.538677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538628    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538707    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538796    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538814    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtpsp\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538817    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.538964    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539246 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.539114    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539246 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.539124    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.539325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.539253    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.541483 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.541455    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.541794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.541773    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.564294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.564259    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.564486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.564458    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtpsp\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.601212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.601180    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:30.617558 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.617530    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:30.961202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.961051    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:17:30.964533 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.964485    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:30.964695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.964584    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:30.964695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.964637    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:30.974257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.972183    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:17:30.978700 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:17:30.978669    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e3bcb6_f16b_4b8d_a9f0_6c5fa7be54d5.slice/crio-6732a6c3b85943ad1102d0221608e2256e83bd4fc07a7254935b43432e66cfe1 WatchSource:0}: Error finding container 6732a6c3b85943ad1102d0221608e2256e83bd4fc07a7254935b43432e66cfe1: Status 404 returned error can't find the container with id 6732a6c3b85943ad1102d0221608e2256e83bd4fc07a7254935b43432e66cfe1
Nov 02 13:17:30.983559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.983516    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:30.983678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.983609    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:30.983678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:30.983659    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:17:31.564379 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.564339    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" event={"ID":"6e5ddc38-0f47-4406-b626-cf38b37cdd43","Type":"ContainerStarted","Data":"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"}
Nov 02 13:17:31.565056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.565023    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" event={"ID":"6e5ddc38-0f47-4406-b626-cf38b37cdd43","Type":"ContainerStarted","Data":"cd99ebdc0994aefbb920617a58d0534490101691814fb8941c7562fbe4e2ea28"}
Nov 02 13:17:31.567656 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.567620    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" event={"ID":"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5","Type":"ContainerStarted","Data":"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"}
Nov 02 13:17:31.567792 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.567676    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" event={"ID":"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5","Type":"ContainerStarted","Data":"6732a6c3b85943ad1102d0221608e2256e83bd4fc07a7254935b43432e66cfe1"}
Nov 02 13:17:31.601376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.601342    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:31.611748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.611684    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podStartSLOduration=1.611664373 podStartE2EDuration="1.611664373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:31.610570326 +0000 UTC m=+3049.259624284" watchObservedRunningTime="2026-11-02 13:17:31.611664373 +0000 UTC m=+3049.260718330"
Nov 02 13:17:31.618063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.618039    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:31.662196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:31.662133    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podStartSLOduration=1.662118956 podStartE2EDuration="1.662118956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:31.65863172 +0000 UTC m=+3049.307685678" watchObservedRunningTime="2026-11-02 13:17:31.662118956 +0000 UTC m=+3049.311172914"
Nov 02 13:17:32.616774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:32.614394    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:32.616774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:32.614457    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:32.653854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:32.636570    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:32.653854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:32.636631    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:33.602431 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:33.602390    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:33.602624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:33.602448    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:33.618271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:33.618229    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:33.618653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:33.618292    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:34.488238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:34.487965    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" podUID="e64567bb-8703-4dba-ae12-50436554814d" containerName="payload-processing" containerID="cri-o://af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0" gracePeriod=25
Nov 02 13:17:34.616738 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:34.616701    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:34.617045 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:34.617010    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:34.636852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:34.623818    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:34.636852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:34.623977    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:35.122180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.120557    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:17:35.268850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.260431    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\") pod \"e64567bb-8703-4dba-ae12-50436554814d\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") "
Nov 02 13:17:35.268850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.260515    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\") pod \"e64567bb-8703-4dba-ae12-50436554814d\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") "
Nov 02 13:17:35.268850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.260666    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\") pod \"e64567bb-8703-4dba-ae12-50436554814d\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") "
Nov 02 13:17:35.268850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.261007    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\") pod \"e64567bb-8703-4dba-ae12-50436554814d\" (UID: \"e64567bb-8703-4dba-ae12-50436554814d\") "
Nov 02 13:17:35.285852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.283062    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume" pod "e64567bb-8703-4dba-ae12-50436554814d" (UID: "e64567bb-8703-4dba-ae12-50436554814d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:35.297258 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.296960    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume" pod "e64567bb-8703-4dba-ae12-50436554814d" (UID: "e64567bb-8703-4dba-ae12-50436554814d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:35.299037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.299012    4928 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-routing-overlay-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:17:35.299187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.299174    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e64567bb-8703-4dba-ae12-50436554814d-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:17:35.300511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.300115    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55" pod "e64567bb-8703-4dba-ae12-50436554814d" (UID: "e64567bb-8703-4dba-ae12-50436554814d"). InnerVolumeSpecName "kube-api-access-65f55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:35.310038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.308911    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials" pod "e64567bb-8703-4dba-ae12-50436554814d" (UID: "e64567bb-8703-4dba-ae12-50436554814d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:35.401843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.400509    4928 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-provider-credentials\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:17:35.401843 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.400537    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65f55\" (UniqueName: \"kubernetes.io/projected/e64567bb-8703-4dba-ae12-50436554814d-kube-api-access-65f55\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:17:35.679967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.653343    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:35.679967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.653877    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:35.679967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.654783    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:35.679967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.654856    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:35.944940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.944556    4928 generic.go:356] "Generic (PLEG): container finished" podID="e64567bb-8703-4dba-ae12-50436554814d" containerID="af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0" exitCode=0
Nov 02 13:17:35.944940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.944678    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" event={"ID":"e64567bb-8703-4dba-ae12-50436554814d","Type":"ContainerDied","Data":"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"}
Nov 02 13:17:35.944940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.944737    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp" event={"ID":"e64567bb-8703-4dba-ae12-50436554814d","Type":"ContainerDied","Data":"9bab23372eea775374426567215b7be4801e25039231440722ee2fed3c660cfb"}
Nov 02 13:17:35.944940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.944773    4928 scope.go:122] "RemoveContainer" containerID="af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"
Nov 02 13:17:35.945282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:35.944971    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"
Nov 02 13:17:36.015040 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.014413    4928 scope.go:122] "RemoveContainer" containerID="af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"
Nov 02 13:17:36.015601 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:17:36.015522    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0\": container with ID starting with af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0 not found: ID does not exist" containerID="af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"
Nov 02 13:17:36.015733 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.015630    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0"} err="failed to get container status \"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0\": rpc error: code = NotFound desc = could not find container \"af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0\": container with ID starting with af100fcd5467c7856b55dcb00319c5d77dae1534042c092d82dfd1b67d93c5c0 not found: ID does not exist"
Nov 02 13:17:36.082851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.081925    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:17:36.085787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.085752    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-86bc9g8sp"]
Nov 02 13:17:36.520535 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.520500    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64567bb-8703-4dba-ae12-50436554814d" path="/var/lib/kubelet/pods/e64567bb-8703-4dba-ae12-50436554814d/volumes"
Nov 02 13:17:36.602049 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.602012    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:36.602226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.602072    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:36.618262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.618230    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:36.618420 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:36.618287    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:37.601883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:37.601799    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 13:17:37.601883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:37.601868    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 13:17:37.618457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:37.618425    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:37.618621 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:37.618477    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:38.606394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:38.606358    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:38.606810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:38.606732    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:38.607601 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:38.607580    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:17:38.618708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:38.618673    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 13:17:38.618845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:38.618742    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 13:17:39.624418 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:39.624383    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:39.624867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:39.624857    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:39.625680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:17:39.625650    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:17:39.731642 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:17:39.731604    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:18:00.967692 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:00.967643    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:18:00.968866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:00.968843    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e64567bb-8703-4dba-ae12-50436554814d" containerName="payload-processing"
Nov 02 13:18:00.968866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:00.968864    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64567bb-8703-4dba-ae12-50436554814d" containerName="payload-processing"
Nov 02 13:18:00.969026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:00.969015    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64567bb-8703-4dba-ae12-50436554814d" containerName="payload-processing"
Nov 02 13:18:00.978162 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:00.978135    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.004260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.004216    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e8a402-serving-cert\""
Nov 02 13:18:01.109906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.091878    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.109906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.091931    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.109906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.091962    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lg7\" (UniqueName: \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.109906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.092041    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.268746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.257404    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.268746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.257815    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.268746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.257861    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.268746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.257887    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lg7\" (UniqueName: \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.271865 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:01.270947    4928 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-serving-cert: secret "maas-api-e2e-inf-a-e8a402-serving-cert" not found
Nov 02 13:18:01.271865 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:01.271027    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls podName:797f5096-ae6a-4454-a351-25751edae1af nodeName:}" failed. No retries permitted until 2026-11-02 13:18:01.77100293 +0000 UTC m=+3079.420056867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls") pod "maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" (UID: "797f5096-ae6a-4454-a351-25751edae1af") : secret "maas-api-e2e-inf-a-e8a402-serving-cert" not found
Nov 02 13:18:01.272577 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.272546    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.309870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.309077    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.398010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.392034    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:18:01.403728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.403638    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lg7\" (UniqueName: \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.775464 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.775423    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.778060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.778034    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") pod \"maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:01.913534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:01.913497    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:02.254195 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:02.254164    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:18:02.254634 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:18:02.254610    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797f5096_ae6a_4454_a351_25751edae1af.slice/crio-8b9bd15b4e60d1ee4d8b6737c13c033810a4ce8735018dd7045665e3756c5d4a WatchSource:0}: Error finding container 8b9bd15b4e60d1ee4d8b6737c13c033810a4ce8735018dd7045665e3756c5d4a: Status 404 returned error can't find the container with id 8b9bd15b4e60d1ee4d8b6737c13c033810a4ce8735018dd7045665e3756c5d4a
Nov 02 13:18:02.500078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:02.499994    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" event={"ID":"797f5096-ae6a-4454-a351-25751edae1af","Type":"ContainerStarted","Data":"8b9bd15b4e60d1ee4d8b6737c13c033810a4ce8735018dd7045665e3756c5d4a"}
Nov 02 13:18:03.503084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:03.503040    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" event={"ID":"797f5096-ae6a-4454-a351-25751edae1af","Type":"ContainerStarted","Data":"99d320a444068816e0147c8a8a33987a39c1e8419845abb12fdc188cf3ac0530"}
Nov 02 13:18:03.503084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:03.503096    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:03.555031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:03.554967    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" podStartSLOduration=3.066102615 podStartE2EDuration="3.554952296s" totalImagesPullingTime="488.849681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:03.549093172 +0000 UTC m=+3081.198147155" watchObservedRunningTime="2026-11-02 13:18:03.554952296 +0000 UTC m=+3081.204006250"
Nov 02 13:18:09.541174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:09.535326    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:18:09.745551 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:09.745505    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:18:15.458129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.458093    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:18:15.463820 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.463739    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.492365 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.492326    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:18:15.583634 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583595    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.583819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583675    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.583819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583745    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.583931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583811    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.583931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583889    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.584004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.583973    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.584039 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.584022    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzr2\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.584137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.584123    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.584259 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.584236    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686349 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686300    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686410    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686509    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686543    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686569    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686738 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686670    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686800 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686752    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzr2\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686819    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.686927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686881    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.687063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.686927    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.687063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.687003    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.687185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.687062    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.687248 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.687230    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.687522 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.687496    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.689154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.689130    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.689316 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.689295    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.711881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.711783    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzr2\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.711881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.711804    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:15.788657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:15.788615    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:16.193293 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.193261    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:18:16.193578 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:18:16.193555    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8cd4fc_9962_49f3_a611_150cc57dac57.slice/crio-f39dd685eb5622917093eda66d10a25340664a7edc57875e7bd0aad20f3067c6 WatchSource:0}: Error finding container f39dd685eb5622917093eda66d10a25340664a7edc57875e7bd0aad20f3067c6: Status 404 returned error can't find the container with id f39dd685eb5622917093eda66d10a25340664a7edc57875e7bd0aad20f3067c6
Nov 02 13:18:16.197004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.196964    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:18:16.197122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.197048    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:18:16.197122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.197092    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:18:16.624455 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.624419    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" event={"ID":"7f8cd4fc-9962-49f3-a611-150cc57dac57","Type":"ContainerStarted","Data":"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"}
Nov 02 13:18:16.625097 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.624487    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" event={"ID":"7f8cd4fc-9962-49f3-a611-150cc57dac57","Type":"ContainerStarted","Data":"f39dd685eb5622917093eda66d10a25340664a7edc57875e7bd0aad20f3067c6"}
Nov 02 13:18:16.682055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.681988    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podStartSLOduration=1.681973577 podStartE2EDuration="1.681973577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:16.675981318 +0000 UTC m=+3094.325035277" watchObservedRunningTime="2026-11-02 13:18:16.681973577 +0000 UTC m=+3094.331027535"
Nov 02 13:18:16.789176 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:16.789142    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:17.790893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:17.790855    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:17.791317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:17.790928    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:18.790001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:18.789962    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:18.790204 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:18.790039    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:19.789889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:19.789816    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:19.790302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:19.789912    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:20.789658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:20.789624    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:20.789883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:20.789690    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:21.789761 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:21.789725    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:21.790204 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:21.789774    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:22.790073 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:22.790026    4928 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:18:22.790581 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:22.790111    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 13:18:23.795175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:23.795086    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:23.795594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:23.795581    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:23.796526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:23.796502    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:18:28.965316 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:28.965278    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:18:28.965776 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:28.965537    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy" containerID="cri-o://648c717d892bc54c93c9b3f321a1fa78e3dc6ebd51e1d4da4ee519266304d0bd" gracePeriod=30
Nov 02 13:18:33.543395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:33.543357    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:18:33.543974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:33.543593    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy" containerID="cri-o://d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94" gracePeriod=30
Nov 02 13:18:34.094066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.094008    4928 generic.go:356] "Generic (PLEG): container finished" podID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerID="648c717d892bc54c93c9b3f321a1fa78e3dc6ebd51e1d4da4ee519266304d0bd" exitCode=0
Nov 02 13:18:34.094229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.094083    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" event={"ID":"f0b70a8b-ad95-4121-8adc-39f7283007ab","Type":"ContainerDied","Data":"648c717d892bc54c93c9b3f321a1fa78e3dc6ebd51e1d4da4ee519266304d0bd"}
Nov 02 13:18:34.379267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.379238    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:18:34.519744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.519707    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.519942 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.519869    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.520001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.519937    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.520086 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520065    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.520145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520128    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.520280 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520255    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520302    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520524    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520248    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520553    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520664    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520702    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\") pod \"f0b70a8b-ad95-4121-8adc-39f7283007ab\" (UID: \"f0b70a8b-ad95-4121-8adc-39f7283007ab\") "
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.520739    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:34.521267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521150    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:34.521816 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521614    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.521816 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521639    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b70a8b-ad95-4121-8adc-39f7283007ab-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.521816 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521653    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.521816 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521678    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.521816 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.521693    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.522931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.522904    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:34.524367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.524337    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:34.524367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.524362    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:34.524731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.524708    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv" pod "f0b70a8b-ad95-4121-8adc-39f7283007ab" (UID: "f0b70a8b-ad95-4121-8adc-39f7283007ab"). InnerVolumeSpecName "kube-api-access-q2hqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:34.623854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.623736    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.623854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.623767    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.623854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.623779    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b70a8b-ad95-4121-8adc-39f7283007ab-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:34.623854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:34.623792    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hqv\" (UniqueName: \"kubernetes.io/projected/f0b70a8b-ad95-4121-8adc-39f7283007ab-kube-api-access-q2hqv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:35.111511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:35.111477    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"
Nov 02 13:18:35.111511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:35.111501    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf" event={"ID":"f0b70a8b-ad95-4121-8adc-39f7283007ab","Type":"ContainerDied","Data":"996196c7efcce88f04027d20b45c3c0c92d4c4a700023e983b6936a1994e2b9c"}
Nov 02 13:18:35.111771 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:35.111572    4928 scope.go:122] "RemoveContainer" containerID="648c717d892bc54c93c9b3f321a1fa78e3dc6ebd51e1d4da4ee519266304d0bd"
Nov 02 13:18:35.175719 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:35.175654    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:18:35.191759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:35.191718    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-785rf"]
Nov 02 13:18:36.499793 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:36.499753    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" path="/var/lib/kubelet/pods/f0b70a8b-ad95-4121-8adc-39f7283007ab/volumes"
Nov 02 13:18:38.903455 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:38.903425    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:18:39.001708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.001664    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.001912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.001725    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.001912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.001846    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.001930    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.001959    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002170 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.002099    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002170 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.002153    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.002274    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.002360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.002321    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\") pod \"b7f57607-fb21-452a-8621-d92cda863242\" (UID: \"b7f57607-fb21-452a-8621-d92cda863242\") "
Nov 02 13:18:39.004182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.004145    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.005356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.004957    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.005356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.005071    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.005356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.005326    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.005950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.005777    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:39.007184 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.007158    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "kube-api-access-4x994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.007763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.007740    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.009953 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.009928    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:39.010272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.010242    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy" pod "b7f57607-fb21-452a-8621-d92cda863242" (UID: "b7f57607-fb21-452a-8621-d92cda863242"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.105485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105444    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x994\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-kube-api-access-4x994\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105484    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105502    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f57607-fb21-452a-8621-d92cda863242-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105526    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f57607-fb21-452a-8621-d92cda863242-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105539    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f57607-fb21-452a-8621-d92cda863242-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105553    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105573    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105587    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.105726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.105613    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f57607-fb21-452a-8621-d92cda863242-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.178395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.178311    4928 generic.go:356] "Generic (PLEG): container finished" podID="b7f57607-fb21-452a-8621-d92cda863242" containerID="d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94" exitCode=0
Nov 02 13:18:39.178395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.178362    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"
Nov 02 13:18:39.178630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.178391    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" event={"ID":"b7f57607-fb21-452a-8621-d92cda863242","Type":"ContainerDied","Data":"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"}
Nov 02 13:18:39.178630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.178445    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6" event={"ID":"b7f57607-fb21-452a-8621-d92cda863242","Type":"ContainerDied","Data":"ad84ff1149b339c36ad2d8b0abd79b850435456e9d4836eed70b387140ecc558"}
Nov 02 13:18:39.178630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.178468    4928 scope.go:122] "RemoveContainer" containerID="d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"
Nov 02 13:18:39.193643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.193619    4928 scope.go:122] "RemoveContainer" containerID="d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"
Nov 02 13:18:39.193976 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:39.193955    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94\": container with ID starting with d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94 not found: ID does not exist" containerID="d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"
Nov 02 13:18:39.194061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.193986    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94"} err="failed to get container status \"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94\": rpc error: code = NotFound desc = could not find container \"d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94\": container with ID starting with d37714c54f8a9dead1a2e2854510d59aedba396e792ad7d21db8ae210d1b8b94 not found: ID does not exist"
Nov 02 13:18:39.248538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.248493    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:18:39.254080 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:39.254039    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-7gqw6"]
Nov 02 13:18:39.747992 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:39.747946    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:18:40.501132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:40.501093    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f57607-fb21-452a-8621-d92cda863242" path="/var/lib/kubelet/pods/b7f57607-fb21-452a-8621-d92cda863242/volumes"
Nov 02 13:18:42.577738 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.577706    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:18:42.578226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.577980    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy" containerID="cri-o://d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b" gracePeriod=30
Nov 02 13:18:42.592180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.592140    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:18:42.592407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.592373    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy" containerID="cri-o://994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef" gracePeriod=30
Nov 02 13:18:42.600728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.600698    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:18:42.600982 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:42.600950    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy" containerID="cri-o://9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2" gracePeriod=30
Nov 02 13:18:47.933992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:47.933963    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:18:47.950332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:47.950300    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:18:48.022940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.022901    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:18:48.076633 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076593    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.076842 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076662    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.076842 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076709    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.076842 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076745    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076845    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076916    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076920    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.077046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.076970    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077033    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077081    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077187    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.077334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077225    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077269    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077343    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077379    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077401    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.077613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077404    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077523    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077669    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\") pod \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\" (UID: \"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9\") "
Nov 02 13:18:48.077920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077715    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.077920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077759    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\") pod \"254972ae-c46e-490a-8d7c-aa10b01b5698\" (UID: \"254972ae-c46e-490a-8d7c-aa10b01b5698\") "
Nov 02 13:18:48.078758 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.078734    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.078897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.078764    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.078897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.078779    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.080215 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.077690    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.078005    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.078345    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.079147    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.079493    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.079907    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:48.080317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.080187    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.083311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.083275    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:48.083642 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.083609    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.085241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.085214    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.085241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.085218    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:48.085390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.085250    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.085433 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.085385    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.087117 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.087089    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x" pod "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" (UID: "a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9"). InnerVolumeSpecName "kube-api-access-46l9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.087368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.087341    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc" pod "254972ae-c46e-490a-8d7c-aa10b01b5698" (UID: "254972ae-c46e-490a-8d7c-aa10b01b5698"). InnerVolumeSpecName "kube-api-access-tlwzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.179900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.179782    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.179927    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180033    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180116    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180193    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180238    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180265    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.180454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180344    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.180454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180397    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180568    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.180675 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180605    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180694    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\") pod \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\" (UID: \"46ec0cc1-444b-4b80-90d3-104c4466f2d9\") "
Nov 02 13:18:48.180802 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.180781    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.181244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181204    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:48.181621 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181581    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181621 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181618    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181638    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181653    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254972ae-c46e-490a-8d7c-aa10b01b5698-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181667    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181683    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181697    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181710    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181725    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181738    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181750    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181763    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254972ae-c46e-490a-8d7c-aa10b01b5698-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181776    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.181786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181791    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181805    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlwzc\" (UniqueName: \"kubernetes.io/projected/254972ae-c46e-490a-8d7c-aa10b01b5698-kube-api-access-tlwzc\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181820    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181853    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46l9x\" (UniqueName: \"kubernetes.io/projected/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9-kube-api-access-46l9x\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181867    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181882    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254972ae-c46e-490a-8d7c-aa10b01b5698-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.182493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.181897    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.183443 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.183408    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:48.183540 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.183463    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:48.184191 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.184164    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "kube-api-access-4dztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.185665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.185616    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token" pod "46ec0cc1-444b-4b80-90d3-104c4466f2d9" (UID: "46ec0cc1-444b-4b80-90d3-104c4466f2d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:48.269659 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.269622    4928 generic.go:356] "Generic (PLEG): container finished" podID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerID="994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef" exitCode=0
Nov 02 13:18:48.269899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.269677    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"
Nov 02 13:18:48.269899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.269674    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" event={"ID":"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9","Type":"ContainerDied","Data":"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"}
Nov 02 13:18:48.269899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.269878    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj" event={"ID":"a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9","Type":"ContainerDied","Data":"3180bfbaae1eff0791502db4086bf469a84928871a1b598d6e4eb5bcd46d0058"}
Nov 02 13:18:48.270089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.269907    4928 scope.go:122] "RemoveContainer" containerID="994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"
Nov 02 13:18:48.271407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.271309    4928 generic.go:356] "Generic (PLEG): container finished" podID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerID="9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2" exitCode=0
Nov 02 13:18:48.271407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.271342    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" event={"ID":"46ec0cc1-444b-4b80-90d3-104c4466f2d9","Type":"ContainerDied","Data":"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"}
Nov 02 13:18:48.271407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.271362    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"
Nov 02 13:18:48.271407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.271394    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs" event={"ID":"46ec0cc1-444b-4b80-90d3-104c4466f2d9","Type":"ContainerDied","Data":"5ab876f010721135b9cd9c81a0b1cbdc9e88f2e5aedf5a5aa4bfda770f0599f7"}
Nov 02 13:18:48.273261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.273219    4928 generic.go:356] "Generic (PLEG): container finished" podID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerID="d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b" exitCode=0
Nov 02 13:18:48.273408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.273269    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"
Nov 02 13:18:48.273408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.273263    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" event={"ID":"254972ae-c46e-490a-8d7c-aa10b01b5698","Type":"ContainerDied","Data":"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"}
Nov 02 13:18:48.273408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.273396    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w" event={"ID":"254972ae-c46e-490a-8d7c-aa10b01b5698","Type":"ContainerDied","Data":"1b85ca3c993be3d6f89e8386b68abf524f047039bbc4916e87b770b7f9569e3e"}
Nov 02 13:18:48.283012 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.282984    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.283127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.283012    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dztr\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-kube-api-access-4dztr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.283127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.283027    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.283127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.283035    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ec0cc1-444b-4b80-90d3-104c4466f2d9-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.283956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.283940    4928 scope.go:122] "RemoveContainer" containerID="994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"
Nov 02 13:18:48.284273 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:48.284255    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef\": container with ID starting with 994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef not found: ID does not exist" containerID="994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"
Nov 02 13:18:48.284328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.284282    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef"} err="failed to get container status \"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef\": rpc error: code = NotFound desc = could not find container \"994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef\": container with ID starting with 994b80b6a2401ad87c5e33af87cfae64ab8bbb4b68b6017af21a829e03c1e7ef not found: ID does not exist"
Nov 02 13:18:48.284328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.284298    4928 scope.go:122] "RemoveContainer" containerID="9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"
Nov 02 13:18:48.295676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.295656    4928 scope.go:122] "RemoveContainer" containerID="9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"
Nov 02 13:18:48.296108 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:48.296087    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2\": container with ID starting with 9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2 not found: ID does not exist" containerID="9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"
Nov 02 13:18:48.296217 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.296118    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2"} err="failed to get container status \"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2\": rpc error: code = NotFound desc = could not find container \"9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2\": container with ID starting with 9d2bbae1831a5b231874ad930beaa319b468e7038a4d76317d96a14f417358b2 not found: ID does not exist"
Nov 02 13:18:48.296217 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.296141    4928 scope.go:122] "RemoveContainer" containerID="d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"
Nov 02 13:18:48.307216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.307195    4928 scope.go:122] "RemoveContainer" containerID="d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"
Nov 02 13:18:48.307566 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:18:48.307532    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b\": container with ID starting with d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b not found: ID does not exist" containerID="d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"
Nov 02 13:18:48.307670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.307576    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b"} err="failed to get container status \"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b\": rpc error: code = NotFound desc = could not find container \"d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b\": container with ID starting with d79412efdba35b9af9d731382d818ce915b6e20212d30bed18c8fe52fe9d190b not found: ID does not exist"
Nov 02 13:18:48.346933 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.346901    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:18:48.357302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.357272    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jwzmj"]
Nov 02 13:18:48.411297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.411226    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:18:48.417627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.417593    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xxzqs"]
Nov 02 13:18:48.452591 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.452491    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:18:48.460881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.460848    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-b2r5w"]
Nov 02 13:18:48.501362 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.501322    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" path="/var/lib/kubelet/pods/254972ae-c46e-490a-8d7c-aa10b01b5698/volumes"
Nov 02 13:18:48.501991 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.501972    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" path="/var/lib/kubelet/pods/46ec0cc1-444b-4b80-90d3-104c4466f2d9/volumes"
Nov 02 13:18:48.502500 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:48.502485    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" path="/var/lib/kubelet/pods/a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9/volumes"
Nov 02 13:18:57.557622 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:57.557571    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 13:18:57.558186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:57.557818    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" containerID="cri-o://91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133" gracePeriod=30
Nov 02 13:18:58.769775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.769737    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:18:58.770587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770568    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy"
Nov 02 13:18:58.770587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770588    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770601    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770607    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770635    4928 cpu_manager.go:414] "containerMap: removing container" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770641    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770649    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770654    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770672    4928 cpu_manager.go:414] "containerMap: removing container" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy"
Nov 02 13:18:58.770728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770676    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy"
Nov 02 13:18:58.771203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770836    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ece68d-f9b2-4b0b-b13b-620b4e2d2cc9" containerName="istio-proxy"
Nov 02 13:18:58.771203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770849    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f57607-fb21-452a-8621-d92cda863242" containerName="istio-proxy"
Nov 02 13:18:58.771203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770862    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="254972ae-c46e-490a-8d7c-aa10b01b5698" containerName="istio-proxy"
Nov 02 13:18:58.771203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770867    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b70a8b-ad95-4121-8adc-39f7283007ab" containerName="istio-proxy"
Nov 02 13:18:58.771203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.770875    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ec0cc1-444b-4b80-90d3-104c4466f2d9" containerName="istio-proxy"
Nov 02 13:18:58.777214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.777189    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.792537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.792504    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:18:58.810009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.809975    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-357b30-openshift-default-dockercfg-cftqn\""
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900294    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900375    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgvf\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900461    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900539    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900593    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.900949    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.901063    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.901111    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:58.904901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:58.901150    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008018    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008119    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008154    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008182    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008332    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008364    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgvf\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008410    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008465    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.008850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.008508    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.009509 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.009040    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.009509 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.009283    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.009635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.009516    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.082038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.038553    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.082038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.062436    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.094860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.090525    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.169208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.169136    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.191884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.170532    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgvf\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.229848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.227556    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\") pod \"e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") " pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.422367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.421548    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:18:59.880306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.880144    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:18:59.880626 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.880576    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:18:59.880739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.880715    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:18:59.880840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:18:59.880804    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:19:00.480851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:00.480053    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerStarted","Data":"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"}
Nov 02 13:19:00.480851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:00.480126    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerStarted","Data":"e6d3be34d665ef3d784d1c11edef3251534b351864e7e61506aa9b9cc3163221"}
Nov 02 13:19:00.555853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:00.555173    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podStartSLOduration=2.555153485 podStartE2EDuration="2.555153485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:00.532330765 +0000 UTC m=+3138.181384724" watchObservedRunningTime="2026-11-02 13:19:00.555153485 +0000 UTC m=+3138.204207445"
Nov 02 13:19:01.422113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.422079    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:19:01.438848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.436147    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:19:01.459852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.459062    4928 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:19:01.459852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.459124    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:19:01.490529 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.490159    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:19:01.491472 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:01.491445    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:19:02.964196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:02.963799    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 13:19:03.003689 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.003658    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.003916 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.003736    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.003916 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.003814    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.003916 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.003869    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004050    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004109    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004245 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004151    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004245 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004184    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004270    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\") pod \"24c018ca-6757-48f1-836b-02cadf9f088f\" (UID: \"24c018ca-6757-48f1-836b-02cadf9f088f\") "
Nov 02 13:19:03.004427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004171    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:03.004427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004261    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:03.004427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004294    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:19:03.004628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004433    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:03.004628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.004517    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:03.005269 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.005236    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c018ca-6757-48f1-836b-02cadf9f088f-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.005405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.005270    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.005405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.005284    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.005405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.005292    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.005405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.005301    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.006489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.006465    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:03.006590 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.006543    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "kube-api-access-gxlpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:03.006706 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.006689    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:03.006770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.006698    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo" pod "24c018ca-6757-48f1-836b-02cadf9f088f" (UID: "24c018ca-6757-48f1-836b-02cadf9f088f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:19:03.114670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.113766    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxlpp\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-kube-api-access-gxlpp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.114670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.113808    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c018ca-6757-48f1-836b-02cadf9f088f-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.114670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.113843    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c018ca-6757-48f1-836b-02cadf9f088f-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.114670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.113857    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c018ca-6757-48f1-836b-02cadf9f088f-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:19:03.508761 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.508677    4928 generic.go:356] "Generic (PLEG): container finished" podID="24c018ca-6757-48f1-836b-02cadf9f088f" containerID="91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133" exitCode=0
Nov 02 13:19:03.508761 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.508727    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"
Nov 02 13:19:03.508958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.508763    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" event={"ID":"24c018ca-6757-48f1-836b-02cadf9f088f","Type":"ContainerDied","Data":"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"}
Nov 02 13:19:03.508958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.508852    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr" event={"ID":"24c018ca-6757-48f1-836b-02cadf9f088f","Type":"ContainerDied","Data":"203d5ae52a5b154f0f343eb85b72568933adf965d0f69bc9383c9a534ac73a90"}
Nov 02 13:19:03.508958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.508883    4928 scope.go:122] "RemoveContainer" containerID="91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"
Nov 02 13:19:03.521528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.521508    4928 scope.go:122] "RemoveContainer" containerID="91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"
Nov 02 13:19:03.521866 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:19:03.521839    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133\": container with ID starting with 91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133 not found: ID does not exist" containerID="91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"
Nov 02 13:19:03.521958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.521875    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133"} err="failed to get container status \"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133\": rpc error: code = NotFound desc = could not find container \"91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133\": container with ID starting with 91360e33da0b16ccdb116de1cfa66f48dc32b32bbcf3f3e0c8928254289a2133 not found: ID does not exist"
Nov 02 13:19:03.583775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.583740    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 13:19:03.598663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:03.598632    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-bkjrr"]
Nov 02 13:19:04.502370 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:04.502332    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" path="/var/lib/kubelet/pods/24c018ca-6757-48f1-836b-02cadf9f088f/volumes"
Nov 02 13:19:07.554457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.554419    4928 generic.go:356] "Generic (PLEG): container finished" podID="7344f928-7931-47b9-b15a-58fc25d6e878" containerID="151c63225a24b8706d6435cc186c4f9960566a28f60ff1120f55c56a2f7156bc" exitCode=0
Nov 02 13:19:07.555026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.554584    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerDied","Data":"151c63225a24b8706d6435cc186c4f9960566a28f60ff1120f55c56a2f7156bc"}
Nov 02 13:19:07.555026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.554977    4928 scope.go:122] "RemoveContainer" containerID="151c63225a24b8706d6435cc186c4f9960566a28f60ff1120f55c56a2f7156bc"
Nov 02 13:19:07.556956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.556913    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:19:07.557085 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.557009    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:19:07.557085 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:07.557043    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:19:08.567164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:08.567124    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerStarted","Data":"d95a5913f326b9898622df240e7cac18d5ad0350ce1dc8066f9217e74041fd97"}
Nov 02 13:19:08.689938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:08.689902    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:19:08.691651 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:08.691613    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:08.691976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:08.691943    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:09.062002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:09.061960    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:19:09.690262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:09.690222    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:09.690697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:09.690285    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:09.762713 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:19:09.762679    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:19:10.690892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:10.690813    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:10.691324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:10.690935    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:11.703087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:11.697205    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:11.703087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:11.697283    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:12.690250 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:12.690208    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:12.690447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:12.690278    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:13.690676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:13.690645    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 13:19:13.691160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:13.691130    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 13:19:14.696279 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:14.696247    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:19:14.697508 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:14.697481    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:19:39.746271 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:19:39.746227    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:19:58.018110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.018025    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:19:58.019064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.019041    4928 cpu_manager.go:414] "containerMap: removing container" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy"
Nov 02 13:19:58.019064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.019064    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy"
Nov 02 13:19:58.019319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.019302    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c018ca-6757-48f1-836b-02cadf9f088f" containerName="istio-proxy"
Nov 02 13:19:58.024711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.024687    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.055163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.055131    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-357b30-serving-cert\""
Nov 02 13:19:58.069213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.063006    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:19:58.102060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.102010    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.102257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.102134    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.102335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.102303    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fz95\" (UniqueName: \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.102393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.102351    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.203733 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.203694    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fz95\" (UniqueName: \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.203981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.203746    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.204064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.204002    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.204064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.204052    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.204894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.204860    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.206428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.206396    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.206537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.206446    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.233855 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.233807    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fz95\" (UniqueName: \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\") pod \"maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.354910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.354252    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:58.678004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.677975    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:19:58.679971 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:19:58.679941    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa0c403_890b_4006_ad32_eac4cb160996.slice/crio-c9d18e292cf058938ad14e365fd5eac13e2f9cb7fd269286dc6ac97be7e46e56 WatchSource:0}: Error finding container c9d18e292cf058938ad14e365fd5eac13e2f9cb7fd269286dc6ac97be7e46e56: Status 404 returned error can't find the container with id c9d18e292cf058938ad14e365fd5eac13e2f9cb7fd269286dc6ac97be7e46e56
Nov 02 13:19:58.681478 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:58.681460    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:19:59.369311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:59.369199    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" event={"ID":"bfa0c403-890b-4006-ad32-eac4cb160996","Type":"ContainerStarted","Data":"aef28d120bd5b941c75294fd9a7dc9c3e1c192df261311da4a1fa9dc31e4f982"}
Nov 02 13:19:59.369311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:59.369298    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" event={"ID":"bfa0c403-890b-4006-ad32-eac4cb160996","Type":"ContainerStarted","Data":"c9d18e292cf058938ad14e365fd5eac13e2f9cb7fd269286dc6ac97be7e46e56"}
Nov 02 13:19:59.369882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:59.369598    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:19:59.418085 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:19:59.418019    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" podStartSLOduration=1.9991105500000002 podStartE2EDuration="2.41799912s" totalImagesPullingTime="418.88857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:59.415170689 +0000 UTC m=+3197.064224646" watchObservedRunningTime="2026-11-02 13:19:59.41799912 +0000 UTC m=+3197.067053079"
Nov 02 13:20:05.384837 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:05.384791    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:20:09.736233 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:20:09.736185    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:20:31.405610 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.405559    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:20:31.410962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.410932    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.434742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.433028    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8962e9-serving-cert\""
Nov 02 13:20:31.473928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.470524    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.473928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.470571    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6c4\" (UniqueName: \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.473928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.470723    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.473928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.470816    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.485854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.481525    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:20:31.572368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.572328    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.572566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.572428    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.572668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.572643    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.572907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.572886    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6c4\" (UniqueName: \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.573439 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.573412    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.575185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.575163    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.575409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.575388    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.598422 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.598391    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6c4\" (UniqueName: \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\") pod \"maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:31.736448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:31.736415    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:32.101920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:32.101887    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:20:32.102093 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:20:32.102072    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b646e44_487a_4ea8_a520_6c7a12f772b4.slice/crio-f3b5d359e91ccc9678f6bf4def1987caf394eb6c7589cc3a708c8dd3d0659848 WatchSource:0}: Error finding container f3b5d359e91ccc9678f6bf4def1987caf394eb6c7589cc3a708c8dd3d0659848: Status 404 returned error can't find the container with id f3b5d359e91ccc9678f6bf4def1987caf394eb6c7589cc3a708c8dd3d0659848
Nov 02 13:20:32.870151 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:32.870108    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" event={"ID":"8b646e44-487a-4ea8-a520-6c7a12f772b4","Type":"ContainerStarted","Data":"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"}
Nov 02 13:20:32.870681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:32.870175    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" event={"ID":"8b646e44-487a-4ea8-a520-6c7a12f772b4","Type":"ContainerStarted","Data":"f3b5d359e91ccc9678f6bf4def1987caf394eb6c7589cc3a708c8dd3d0659848"}
Nov 02 13:20:32.871077 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:32.871057    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:32.925716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:32.925647    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" podStartSLOduration=1.547550422 podStartE2EDuration="1.925632878s" totalImagesPullingTime="378.082456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:20:32.923535721 +0000 UTC m=+3230.572589681" watchObservedRunningTime="2026-11-02 13:20:32.925632878 +0000 UTC m=+3230.574686836"
Nov 02 13:20:35.171741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:35.171700    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:20:35.172307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:35.171960    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy" containerID="cri-o://515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2" gracePeriod=30
Nov 02 13:20:39.732958 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:20:39.732892    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:20:39.893902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:39.893870    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:20:40.473377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.473312    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:20:40.539196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539155    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539266    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539315    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539405    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539468    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539638 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539595    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539638 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539632    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539684    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.539748 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.539714    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\") pod \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\" (UID: \"c0e41d50-b9be-47bb-a01f-5f0b631596bb\") "
Nov 02 13:20:40.540066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.540037    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:20:40.540066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.540053    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:20:40.540308 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.540285    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:20:40.540387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.540326    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:20:40.540976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.540951    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:40.541385 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.541362    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.541512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.541499    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.541610 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.541599    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.541703 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.541690    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.541836 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.541778    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.545444 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.545400    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "kube-api-access-kfhxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:40.545682 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.545657    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:20:40.545804 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.545696    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:40.560289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.560260    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo" pod "c0e41d50-b9be-47bb-a01f-5f0b631596bb" (UID: "c0e41d50-b9be-47bb-a01f-5f0b631596bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:20:40.643582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.643543    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.643582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.643582    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/c0e41d50-b9be-47bb-a01f-5f0b631596bb-kube-api-access-kfhxh\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.643805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.643599    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.643805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.643613    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e41d50-b9be-47bb-a01f-5f0b631596bb-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:20:40.943550 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.943506    4928 generic.go:356] "Generic (PLEG): container finished" podID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerID="515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2" exitCode=0
Nov 02 13:20:40.944084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.943566    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"
Nov 02 13:20:40.944084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.943796    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerDied","Data":"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"}
Nov 02 13:20:40.944084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.943906    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps" event={"ID":"c0e41d50-b9be-47bb-a01f-5f0b631596bb","Type":"ContainerDied","Data":"0ee5ee02e69776955dcaaf3dd29da1f32edece34fc285a42dafe34e0c0c5a816"}
Nov 02 13:20:40.944084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.943940    4928 scope.go:122] "RemoveContainer" containerID="515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"
Nov 02 13:20:40.958292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.958266    4928 scope.go:122] "RemoveContainer" containerID="e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"
Nov 02 13:20:40.970498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.970474    4928 scope.go:122] "RemoveContainer" containerID="515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"
Nov 02 13:20:40.970840 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:20:40.970799    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2\": container with ID starting with 515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2 not found: ID does not exist" containerID="515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"
Nov 02 13:20:40.970937 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.970850    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2"} err="failed to get container status \"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2\": rpc error: code = NotFound desc = could not find container \"515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2\": container with ID starting with 515732a6b283132ac6800f0bbc104bdcf370b521dd988093a03a7047953710b2 not found: ID does not exist"
Nov 02 13:20:40.970937 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.970876    4928 scope.go:122] "RemoveContainer" containerID="e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"
Nov 02 13:20:40.971200 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:20:40.971172    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3\": container with ID starting with e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3 not found: ID does not exist" containerID="e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"
Nov 02 13:20:40.971272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:40.971211    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3"} err="failed to get container status \"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3\": rpc error: code = NotFound desc = could not find container \"e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3\": container with ID starting with e1498266748d721e3784a0fb027da8148e94495925bac9152c1429fc36e844a3 not found: ID does not exist"
Nov 02 13:20:41.012887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:41.012815    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:20:41.017443 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:41.017414    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bbb7d0-openshift-default-5dcffb6bdf-h7lps"]
Nov 02 13:20:42.500217 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:20:42.500181    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" path="/var/lib/kubelet/pods/c0e41d50-b9be-47bb-a01f-5f0b631596bb/volumes"
Nov 02 13:21:02.531032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:02.530982    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:21:02.531571 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:02.531069    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:21:06.283844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.283785    4928 generic.go:356] "Generic (PLEG): container finished" podID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerID="168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108" exitCode=0
Nov 02 13:21:06.284289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.283855    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerDied","Data":"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"}
Nov 02 13:21:06.284386 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.284367    4928 scope.go:122] "RemoveContainer" containerID="168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"
Nov 02 13:21:06.286554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.286519    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:21:06.286631 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.286609    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:21:06.286673 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:06.286645    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:21:07.293907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:07.293870    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerStarted","Data":"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"}
Nov 02 13:21:07.422128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:07.422096    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:21:07.422329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:07.422304    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:07.422386 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:07.422361    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:08.422185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:08.422143    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:08.422653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:08.422218    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:09.422718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:09.422681    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:09.423233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:09.422760    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:09.764343 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:21:09.764300    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:21:10.422032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:10.421995    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:10.422214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:10.422057    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:11.422057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:11.422019    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:11.422556 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:11.422096    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:12.422144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:12.422105    4928 patch_prober.go:29] interesting pod/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:21:12.422580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:12.422198    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 13:21:13.429124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:13.429087    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:21:13.429681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:13.429657    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:21:13.430356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:13.430336    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:21:34.334383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:34.334344    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:21:34.334910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:34.334602    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy" containerID="cri-o://57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc" gracePeriod=30
Nov 02 13:21:39.641034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.640998    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:21:39.741535 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:21:39.741493    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:21:39.777352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.777311    4928 generic.go:356] "Generic (PLEG): container finished" podID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerID="57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc" exitCode=0
Nov 02 13:21:39.777572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.777412    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"
Nov 02 13:21:39.777572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.777447    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerDied","Data":"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"}
Nov 02 13:21:39.777572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.777504    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls" event={"ID":"ed6dfe07-9efb-4437-a111-d30aabccf018","Type":"ContainerDied","Data":"149464ffe756803092a382c7b1a507fe93456a8ddd242c6b16058d6658da5b32"}
Nov 02 13:21:39.777572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.777529    4928 scope.go:122] "RemoveContainer" containerID="57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"
Nov 02 13:21:39.789755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.789731    4928 scope.go:122] "RemoveContainer" containerID="314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"
Nov 02 13:21:39.800919 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.800875    4928 scope.go:122] "RemoveContainer" containerID="57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"
Nov 02 13:21:39.801247 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:21:39.801224    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc\": container with ID starting with 57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc not found: ID does not exist" containerID="57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"
Nov 02 13:21:39.801336 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801256    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc"} err="failed to get container status \"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc\": rpc error: code = NotFound desc = could not find container \"57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc\": container with ID starting with 57f468ba5b33efc9ef0bc7e76aa5d5302a9f78675397139cc8ee6144fbbdc6dc not found: ID does not exist"
Nov 02 13:21:39.801336 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801277    4928 scope.go:122] "RemoveContainer" containerID="314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"
Nov 02 13:21:39.801440 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801384    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801440 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801428    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801478    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801507    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801531    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801616 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:21:39.801592    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7\": container with ID starting with 314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7 not found: ID does not exist" containerID="314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"
Nov 02 13:21:39.801698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801623    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7"} err="failed to get container status \"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7\": rpc error: code = NotFound desc = could not find container \"314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7\": container with ID starting with 314b23e1337d6ec3d34fa976bce7a9979bc565a00ef9c7ec5a651239796c8bd7 not found: ID does not exist"
Nov 02 13:21:39.801755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801699    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801872 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801851    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.801971 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801868    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.802036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.801969    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.802036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.802022    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:39.802145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.802046    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\") pod \"ed6dfe07-9efb-4437-a111-d30aabccf018\" (UID: \"ed6dfe07-9efb-4437-a111-d30aabccf018\") "
Nov 02 13:21:39.802254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.802227    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.802342 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.802255    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.802496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.802473    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.803420 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.803396    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.803530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.803427    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.803530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.803446    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6dfe07-9efb-4437-a111-d30aabccf018-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.803530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.803462    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.803530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.803476    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.804065 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.804044    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:39.804140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.804098    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.804140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.804102    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:21:39.804351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.804332    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44" pod "ed6dfe07-9efb-4437-a111-d30aabccf018" (UID: "ed6dfe07-9efb-4437-a111-d30aabccf018"). InnerVolumeSpecName "kube-api-access-7dc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:39.904950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.904865    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.904950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.904895    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.904950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.904904    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dc44\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-kube-api-access-7dc44\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.904950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:39.904914    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6dfe07-9efb-4437-a111-d30aabccf018-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:21:40.150513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:40.150482    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:21:40.163598 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:40.163519    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-xf7ls"]
Nov 02 13:21:40.500316 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:40.500275    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" path="/var/lib/kubelet/pods/ed6dfe07-9efb-4437-a111-d30aabccf018/volumes"
Nov 02 13:21:43.245409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:43.245377    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:21:43.255810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:43.255785    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:21:49.405435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:49.405395    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:21:49.439860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:49.439815    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:21:54.406727 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:54.406671    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing" containerID="cri-o://c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40" gracePeriod=25
Nov 02 13:21:54.440311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:54.440263    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing" containerID="cri-o://c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658" gracePeriod=25
Nov 02 13:21:58.689018 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:58.688965    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:21:58.689769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:21:58.689740    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:08.699418 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.699373    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:08.699966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.699384    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:08.954879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.954781    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:22:08.955345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.955309    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" containerID="cri-o://b131a685e8268a2784455494cb3933f973ce6008d46435f5eeec3be1f8d50d4e" gracePeriod=30
Nov 02 13:22:08.964880 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.964814    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:22:08.965112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:08.965062    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" containerID="cri-o://67eb11af8f2ccb6335dfdae848d960ce6ec67e57eefefff8008db526ef9e14cc" gracePeriod=30
Nov 02 13:22:09.730751 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:09.730709    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:22:10.327360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:10.327321    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:10.327566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:10.327394    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:11.789796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:11.789757    4928 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:11.790222 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:11.789817    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:14.260593 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.260544    4928 generic.go:356] "Generic (PLEG): container finished" podID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerID="b131a685e8268a2784455494cb3933f973ce6008d46435f5eeec3be1f8d50d4e" exitCode=0
Nov 02 13:22:14.261294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.261264    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerDied","Data":"b131a685e8268a2784455494cb3933f973ce6008d46435f5eeec3be1f8d50d4e"}
Nov 02 13:22:14.261526 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.261509    4928 scope.go:122] "RemoveContainer" containerID="6e713a17cfffbe5a27ca13cd2efc0d22b3e842a6b1a3e1b75225657e66985a46"
Nov 02 13:22:14.271084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.271054    4928 generic.go:356] "Generic (PLEG): container finished" podID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerID="67eb11af8f2ccb6335dfdae848d960ce6ec67e57eefefff8008db526ef9e14cc" exitCode=0
Nov 02 13:22:14.271303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.271282    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" event={"ID":"398d8f4d-80fe-4e36-b701-2468d193bac8","Type":"ContainerDied","Data":"67eb11af8f2ccb6335dfdae848d960ce6ec67e57eefefff8008db526ef9e14cc"}
Nov 02 13:22:14.313046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.313014    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:22:14.404866 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.404819    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:22:14.413021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.412799    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.412879    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.412956    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413063    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413085    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413142    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413156    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413181    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413271    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413353    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\") pod \"9d3262cb-20d2-473c-9e48-49bc20b0b560\" (UID: \"9d3262cb-20d2-473c-9e48-49bc20b0b560\") "
Nov 02 13:22:14.413756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.413561    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.414180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.414156    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.414247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.414189    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.414881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.414799    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.415113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.415082    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.415184 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.415098    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.417307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.417204    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.417307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.417274    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:14.418075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.418048    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.418874 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.418811    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf" pod "9d3262cb-20d2-473c-9e48-49bc20b0b560" (UID: "9d3262cb-20d2-473c-9e48-49bc20b0b560"). InnerVolumeSpecName "kube-api-access-qtgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.517329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517242    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517521 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517481    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517610 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517518    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517610 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517595    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517645    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517709    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517770    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.517911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517841    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.518056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.517928    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\") pod \"398d8f4d-80fe-4e36-b701-2468d193bac8\" (UID: \"398d8f4d-80fe-4e36-b701-2468d193bac8\") "
Nov 02 13:22:14.518163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518138    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.518679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518656    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518679    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518693    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3262cb-20d2-473c-9e48-49bc20b0b560-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518706    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518716    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518725    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518733    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgvf\" (UniqueName: \"kubernetes.io/projected/9d3262cb-20d2-473c-9e48-49bc20b0b560-kube-api-access-qtgvf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.518920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.518743    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3262cb-20d2-473c-9e48-49bc20b0b560-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.519486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.519456    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.520089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.519719    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.520089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.519904    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.520406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.520378    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.520897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.520869    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:14.522301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.522263    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.524013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.523969    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.524120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.523976    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b" pod "398d8f4d-80fe-4e36-b701-2468d193bac8" (UID: "398d8f4d-80fe-4e36-b701-2468d193bac8"). InnerVolumeSpecName "kube-api-access-lbp4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.620489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620452    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620490    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620504    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620512    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620520    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620528    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398d8f4d-80fe-4e36-b701-2468d193bac8-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620535    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.620716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.620545    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbp4b\" (UniqueName: \"kubernetes.io/projected/398d8f4d-80fe-4e36-b701-2468d193bac8-kube-api-access-lbp4b\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.723721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.723685    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:22:14.789778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.789748    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:22:14.823605 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.823569    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\" (UniqueName: \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\") pod \"78f79307-ad18-4efa-bfe4-13d23e590c56\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") "
Nov 02 13:22:14.823760 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.823680    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\" (UniqueName: \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\") pod \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") "
Nov 02 13:22:14.823901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.823788    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\") pod \"78f79307-ad18-4efa-bfe4-13d23e590c56\" (UID: \"78f79307-ad18-4efa-bfe4-13d23e590c56\") "
Nov 02 13:22:14.824034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.824013    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\") pod \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\" (UID: \"ffcab597-c6c0-4577-b9fa-a1e20c567b56\") "
Nov 02 13:22:14.824430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.824389    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume" pod "78f79307-ad18-4efa-bfe4-13d23e590c56" (UID: "78f79307-ad18-4efa-bfe4-13d23e590c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.824582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.824433    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume" pod "ffcab597-c6c0-4577-b9fa-a1e20c567b56" (UID: "ffcab597-c6c0-4577-b9fa-a1e20c567b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.824761 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.824733    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcab597-c6c0-4577-b9fa-a1e20c567b56-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.824881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.824766    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f79307-ad18-4efa-bfe4-13d23e590c56-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.827225 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.827200    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7" pod "ffcab597-c6c0-4577-b9fa-a1e20c567b56" (UID: "ffcab597-c6c0-4577-b9fa-a1e20c567b56"). InnerVolumeSpecName "kube-api-access-7p4g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.827340 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.827299    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h" pod "78f79307-ad18-4efa-bfe4-13d23e590c56" (UID: "78f79307-ad18-4efa-bfe4-13d23e590c56"). InnerVolumeSpecName "kube-api-access-lgs7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.926208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.926166    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4g7\" (UniqueName: \"kubernetes.io/projected/ffcab597-c6c0-4577-b9fa-a1e20c567b56-kube-api-access-7p4g7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.926208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:14.926201    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgs7h\" (UniqueName: \"kubernetes.io/projected/78f79307-ad18-4efa-bfe4-13d23e590c56-kube-api-access-lgs7h\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:15.281734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.281698    4928 generic.go:356] "Generic (PLEG): container finished" podID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerID="c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40" exitCode=0
Nov 02 13:22:15.282238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.281739    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"
Nov 02 13:22:15.282238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.281783    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" event={"ID":"78f79307-ad18-4efa-bfe4-13d23e590c56","Type":"ContainerDied","Data":"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"}
Nov 02 13:22:15.282238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.281849    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf" event={"ID":"78f79307-ad18-4efa-bfe4-13d23e590c56","Type":"ContainerDied","Data":"639e1464fdd85f4101e7796685136dfaff7f9c344071537c14d5474203f11f87"}
Nov 02 13:22:15.282238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.281876    4928 scope.go:122] "RemoveContainer" containerID="c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"
Nov 02 13:22:15.283600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.283582    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"
Nov 02 13:22:15.283742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.283635    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x" event={"ID":"9d3262cb-20d2-473c-9e48-49bc20b0b560","Type":"ContainerDied","Data":"80fb6a8e66262f0c73528c5708a8f39ed430e8f377adc53a8c45db369ad90a20"}
Nov 02 13:22:15.285519 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.285481    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"
Nov 02 13:22:15.285644 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.285515    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9" event={"ID":"398d8f4d-80fe-4e36-b701-2468d193bac8","Type":"ContainerDied","Data":"06b57f43cc89e8293aeb8da243a1856b74303667b374a44c0193a2664fea10ac"}
Nov 02 13:22:15.287365 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.287341    4928 generic.go:356] "Generic (PLEG): container finished" podID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerID="c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658" exitCode=0
Nov 02 13:22:15.287454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.287417    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"
Nov 02 13:22:15.287512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.287495    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" event={"ID":"ffcab597-c6c0-4577-b9fa-a1e20c567b56","Type":"ContainerDied","Data":"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"}
Nov 02 13:22:15.287553 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.287541    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2" event={"ID":"ffcab597-c6c0-4577-b9fa-a1e20c567b56","Type":"ContainerDied","Data":"4fb3530b07cf3d807292acddded7eef0a7e69f32e5efdfd591880cd307e779fc"}
Nov 02 13:22:15.295108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.295093    4928 scope.go:122] "RemoveContainer" containerID="c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"
Nov 02 13:22:15.295415 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:15.295383    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40\": container with ID starting with c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40 not found: ID does not exist" containerID="c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"
Nov 02 13:22:15.295515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.295423    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40"} err="failed to get container status \"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40\": rpc error: code = NotFound desc = could not find container \"c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40\": container with ID starting with c826d0565b8c6e76777a9773982a230c062d12581bd7867d1612a2416ec34a40 not found: ID does not exist"
Nov 02 13:22:15.295515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.295443    4928 scope.go:122] "RemoveContainer" containerID="b131a685e8268a2784455494cb3933f973ce6008d46435f5eeec3be1f8d50d4e"
Nov 02 13:22:15.306408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.306328    4928 scope.go:122] "RemoveContainer" containerID="67eb11af8f2ccb6335dfdae848d960ce6ec67e57eefefff8008db526ef9e14cc"
Nov 02 13:22:15.319624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.319605    4928 scope.go:122] "RemoveContainer" containerID="c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"
Nov 02 13:22:15.334671 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.334646    4928 scope.go:122] "RemoveContainer" containerID="c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"
Nov 02 13:22:15.335445 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:15.335415    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658\": container with ID starting with c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658 not found: ID does not exist" containerID="c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"
Nov 02 13:22:15.335542 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.335452    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658"} err="failed to get container status \"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658\": rpc error: code = NotFound desc = could not find container \"c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658\": container with ID starting with c9ed46a4a3baee720985406b41f72dbc2da4bf2e12f9298946cb7364ea5eb658 not found: ID does not exist"
Nov 02 13:22:15.362130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.362105    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:22:15.370026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.369607    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6563ed-56f84d78c-7nbmf"]
Nov 02 13:22:15.450414 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.450376    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:22:15.454545 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.454517    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-dkt6x"]
Nov 02 13:22:15.508861 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.508817    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:22:15.514221 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.514190    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6563ed-544d484d6c-mqhj2"]
Nov 02 13:22:15.554513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.554432    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:22:15.560717 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:15.560685    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-p76h9"]
Nov 02 13:22:16.499720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:16.499685    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" path="/var/lib/kubelet/pods/398d8f4d-80fe-4e36-b701-2468d193bac8/volumes"
Nov 02 13:22:16.500285 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:16.500267    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" path="/var/lib/kubelet/pods/78f79307-ad18-4efa-bfe4-13d23e590c56/volumes"
Nov 02 13:22:16.500744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:16.500730    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" path="/var/lib/kubelet/pods/9d3262cb-20d2-473c-9e48-49bc20b0b560/volumes"
Nov 02 13:22:16.501525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:16.501508    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" path="/var/lib/kubelet/pods/ffcab597-c6c0-4577-b9fa-a1e20c567b56/volumes"
Nov 02 13:22:30.093991 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.093946    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:22:30.094632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.094221    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" podUID="04c637e6-143b-4813-b498-49e8cded4614" containerName="maas-api" containerID="cri-o://938ba0540d5be2fe2477de354c238edd3ec0bb530e1e2db6571f762dfc6ef9a8" gracePeriod=30
Nov 02 13:22:30.675860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.675780    4928 generic.go:356] "Generic (PLEG): container finished" podID="04c637e6-143b-4813-b498-49e8cded4614" containerID="938ba0540d5be2fe2477de354c238edd3ec0bb530e1e2db6571f762dfc6ef9a8" exitCode=0
Nov 02 13:22:30.708898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.695608    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" event={"ID":"04c637e6-143b-4813-b498-49e8cded4614","Type":"ContainerDied","Data":"938ba0540d5be2fe2477de354c238edd3ec0bb530e1e2db6571f762dfc6ef9a8"}
Nov 02 13:22:30.777858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.777796    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:22:30.810244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.808435    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\" (UniqueName: \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\") pod \"04c637e6-143b-4813-b498-49e8cded4614\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") "
Nov 02 13:22:30.810244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.808559    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\") pod \"04c637e6-143b-4813-b498-49e8cded4614\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") "
Nov 02 13:22:30.810244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.808638    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\") pod \"04c637e6-143b-4813-b498-49e8cded4614\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") "
Nov 02 13:22:30.810244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.808768    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") pod \"04c637e6-143b-4813-b498-49e8cded4614\" (UID: \"04c637e6-143b-4813-b498-49e8cded4614\") "
Nov 02 13:22:30.818752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.818031    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca" pod "04c637e6-143b-4813-b498-49e8cded4614" (UID: "04c637e6-143b-4813-b498-49e8cded4614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:30.820965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.820896    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls" pod "04c637e6-143b-4813-b498-49e8cded4614" (UID: "04c637e6-143b-4813-b498-49e8cded4614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:30.820965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.820925    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs" pod "04c637e6-143b-4813-b498-49e8cded4614" (UID: "04c637e6-143b-4813-b498-49e8cded4614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:30.821308 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.821277    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9" pod "04c637e6-143b-4813-b498-49e8cded4614" (UID: "04c637e6-143b-4813-b498-49e8cded4614"). InnerVolumeSpecName "kube-api-access-9l9l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:30.910702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.910667    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:30.910702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.910699    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9l9\" (UniqueName: \"kubernetes.io/projected/04c637e6-143b-4813-b498-49e8cded4614-kube-api-access-9l9l9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:30.910702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.910709    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c637e6-143b-4813-b498-49e8cded4614-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:30.910993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:30.910717    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c637e6-143b-4813-b498-49e8cded4614-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:31.686238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:31.686202    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"
Nov 02 13:22:31.686238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:31.686228    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q" event={"ID":"04c637e6-143b-4813-b498-49e8cded4614","Type":"ContainerDied","Data":"a52617748872f27a4b03175a17c785b623b02f4b52d607c41fccc54162360945"}
Nov 02 13:22:31.686769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:31.686310    4928 scope.go:122] "RemoveContainer" containerID="938ba0540d5be2fe2477de354c238edd3ec0bb530e1e2db6571f762dfc6ef9a8"
Nov 02 13:22:31.749294 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:31.749262    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:22:31.753217 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:31.753193    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6563ed-589bfdd84c-8f42q"]
Nov 02 13:22:32.295658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.295617    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:22:32.297901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.295952    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy" containerID="cri-o://f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289" gracePeriod=30
Nov 02 13:22:32.306211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.306178    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:22:32.306471 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.306425    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy" containerID="cri-o://5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a" gracePeriod=30
Nov 02 13:22:32.310437 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.310411    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:22:32.310627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.310594    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy" containerID="cri-o://c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120" gracePeriod=30
Nov 02 13:22:32.520397 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:32.520357    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c637e6-143b-4813-b498-49e8cded4614" path="/var/lib/kubelet/pods/04c637e6-143b-4813-b498-49e8cded4614/volumes"
Nov 02 13:22:37.669329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.669300    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:22:37.696402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.696368    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:22:37.709541 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.709509    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.709712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.709590    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.709712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.709702    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.709896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.709865    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.709966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.709893    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710009    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710045    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710081    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710110    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710248 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710207    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\") pod \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\" (UID: \"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5\") "
Nov 02 13:22:37.710971 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.710946    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.711226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.711204    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.712290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.711904    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.712290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.712251    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.712506 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.712483    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.714249 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.714044    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.715348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.715324    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "kube-api-access-qtpsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.715477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.715383    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.717075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.717053    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy" pod "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" (UID: "91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.719185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.719162    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.746887    4928 generic.go:356] "Generic (PLEG): container finished" podID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerID="c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120" exitCode=0
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.747181    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" event={"ID":"7f8cd4fc-9962-49f3-a611-150cc57dac57","Type":"ContainerDied","Data":"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.747275    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2" event={"ID":"7f8cd4fc-9962-49f3-a611-150cc57dac57","Type":"ContainerDied","Data":"f39dd685eb5622917093eda66d10a25340664a7edc57875e7bd0aad20f3067c6"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.747309    4928 scope.go:122] "RemoveContainer" containerID="c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.747482    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.750182    4928 generic.go:356] "Generic (PLEG): container finished" podID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerID="5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a" exitCode=0
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.750205    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" event={"ID":"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5","Type":"ContainerDied","Data":"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.750253    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9" event={"ID":"91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5","Type":"ContainerDied","Data":"6732a6c3b85943ad1102d0221608e2256e83bd4fc07a7254935b43432e66cfe1"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.751897    4928 generic.go:356] "Generic (PLEG): container finished" podID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerID="f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289" exitCode=0
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.751943    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.751946    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" event={"ID":"6e5ddc38-0f47-4406-b626-cf38b37cdd43","Type":"ContainerDied","Data":"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.752054    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l" event={"ID":"6e5ddc38-0f47-4406-b626-cf38b37cdd43","Type":"ContainerDied","Data":"cd99ebdc0994aefbb920617a58d0534490101691814fb8941c7562fbe4e2ea28"}
Nov 02 13:22:37.752856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.752102    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"
Nov 02 13:22:37.786451 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.786424    4928 scope.go:122] "RemoveContainer" containerID="c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"
Nov 02 13:22:37.786892 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:37.786867    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120\": container with ID starting with c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120 not found: ID does not exist" containerID="c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"
Nov 02 13:22:37.787068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.786906    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120"} err="failed to get container status \"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120\": rpc error: code = NotFound desc = could not find container \"c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120\": container with ID starting with c0a7a23b2d0b2d4973efffb022293ee31e0a818377f92851379305514cbbd120 not found: ID does not exist"
Nov 02 13:22:37.787068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.786928    4928 scope.go:122] "RemoveContainer" containerID="5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"
Nov 02 13:22:37.809482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.809457    4928 scope.go:122] "RemoveContainer" containerID="5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"
Nov 02 13:22:37.809870 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:37.809839    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a\": container with ID starting with 5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a not found: ID does not exist" containerID="5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"
Nov 02 13:22:37.809983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.809880    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a"} err="failed to get container status \"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a\": rpc error: code = NotFound desc = could not find container \"5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a\": container with ID starting with 5aaa87d9876de4ecdd5fa7ff06fc27db2c13b4f9e1880e20d06683b4aa8c0c2a not found: ID does not exist"
Nov 02 13:22:37.809983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.809898    4928 scope.go:122] "RemoveContainer" containerID="f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"
Nov 02 13:22:37.812531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812501    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.812643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812554    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.812643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812593    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.812773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812675    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.812773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812747    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.812905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812782    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.812905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812783    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.812905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812816    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.812905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812887    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.813147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.812956    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.813147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813043    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.813147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813115    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.813147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813142    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.813323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813179    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.813323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813218    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.813323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813288    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.813484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813340    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.813484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813415    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.813611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813493    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\") pod \"7f8cd4fc-9962-49f3-a611-150cc57dac57\" (UID: \"7f8cd4fc-9962-49f3-a611-150cc57dac57\") "
Nov 02 13:22:37.813611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.813553    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\") pod \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\" (UID: \"6e5ddc38-0f47-4406-b626-cf38b37cdd43\") "
Nov 02 13:22:37.814418 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814394    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814422    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814437    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814451    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814466    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814479    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814492    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814504    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814517    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtpsp\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-kube-api-access-qtpsp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.814534 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814532    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814720    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814768    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.814975    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815020    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815353    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815504    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815599    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815642    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.816068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.815730    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.817606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.817582    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.817702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.817628    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.817919 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.817891    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.818199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.818169    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "kube-api-access-lkzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.818807 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.818777    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx" pod "6e5ddc38-0f47-4406-b626-cf38b37cdd43" (UID: "6e5ddc38-0f47-4406-b626-cf38b37cdd43"). InnerVolumeSpecName "kube-api-access-cvckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.820600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.820565    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.820703 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.820670    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo" pod "7f8cd4fc-9962-49f3-a611-150cc57dac57" (UID: "7f8cd4fc-9962-49f3-a611-150cc57dac57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.823705 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.823685    4928 scope.go:122] "RemoveContainer" containerID="f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"
Nov 02 13:22:37.824054 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:37.824030    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289\": container with ID starting with f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289 not found: ID does not exist" containerID="f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"
Nov 02 13:22:37.824154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.824065    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289"} err="failed to get container status \"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289\": rpc error: code = NotFound desc = could not find container \"f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289\": container with ID starting with f5869f33d99d604cae7294a85373dd62a862f84374f4eddf4171a05e68729289 not found: ID does not exist"
Nov 02 13:22:37.886672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.886588    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:22:37.891912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.891882    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-dbpl9"]
Nov 02 13:22:37.915760 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915730    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.915760 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915758    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915771    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915785    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915799    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915809    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915820    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915847    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvckx\" (UniqueName: \"kubernetes.io/projected/6e5ddc38-0f47-4406-b626-cf38b37cdd43-kube-api-access-cvckx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915859    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915871    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8cd4fc-9962-49f3-a611-150cc57dac57-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915882    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915893    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkzr2\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-kube-api-access-lkzr2\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915905    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915917    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915927    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8cd4fc-9962-49f3-a611-150cc57dac57-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.916024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:37.915939    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5ddc38-0f47-4406-b626-cf38b37cdd43-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.108216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.108171    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:22:38.114871 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.114817    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-wzxl2"]
Nov 02 13:22:38.150936 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.150816    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:22:38.154700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.154668    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-vtq2l"]
Nov 02 13:22:38.500165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.500130    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" path="/var/lib/kubelet/pods/6e5ddc38-0f47-4406-b626-cf38b37cdd43/volumes"
Nov 02 13:22:38.500756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.500740    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" path="/var/lib/kubelet/pods/7f8cd4fc-9962-49f3-a611-150cc57dac57/volumes"
Nov 02 13:22:38.501317 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:38.501294    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" path="/var/lib/kubelet/pods/91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5/volumes"
Nov 02 13:22:39.578166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:39.578131    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:22:39.748425 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:39.748384    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:22:42.722809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:42.722771    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:22:42.723335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:42.722979    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" podUID="d95d9c1b-9742-4173-add0-df61d21600a9" containerName="maas-api" containerID="cri-o://aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5" gracePeriod=30
Nov 02 13:22:43.064253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.061036    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:22:43.237914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.237548    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\") pod \"d95d9c1b-9742-4173-add0-df61d21600a9\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") "
Nov 02 13:22:43.237914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.237656    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\") pod \"d95d9c1b-9742-4173-add0-df61d21600a9\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") "
Nov 02 13:22:43.237914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.237703    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") pod \"d95d9c1b-9742-4173-add0-df61d21600a9\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") "
Nov 02 13:22:43.237914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.237731    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\" (UniqueName: \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\") pod \"d95d9c1b-9742-4173-add0-df61d21600a9\" (UID: \"d95d9c1b-9742-4173-add0-df61d21600a9\") "
Nov 02 13:22:43.246416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.246374    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca" pod "d95d9c1b-9742-4173-add0-df61d21600a9" (UID: "d95d9c1b-9742-4173-add0-df61d21600a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:43.299054 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.299012    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls" pod "d95d9c1b-9742-4173-add0-df61d21600a9" (UID: "d95d9c1b-9742-4173-add0-df61d21600a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:43.368850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.341532    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d9c1b-9742-4173-add0-df61d21600a9-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:43.368850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.341576    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:43.393908 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.372569    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42" pod "d95d9c1b-9742-4173-add0-df61d21600a9" (UID: "d95d9c1b-9742-4173-add0-df61d21600a9"). InnerVolumeSpecName "kube-api-access-f5m42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:43.393908 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.386884    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401383    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401418    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401439    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d95d9c1b-9742-4173-add0-df61d21600a9" containerName="maas-api"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401447    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95d9c1b-9742-4173-add0-df61d21600a9" containerName="maas-api"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401474    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401484    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401496    4928 cpu_manager.go:414] "containerMap: removing container" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401503    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401519    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401527    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401552    4928 cpu_manager.go:414] "containerMap: removing container" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401568    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401580    4928 cpu_manager.go:414] "containerMap: removing container" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401587    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401599    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401605    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401626    4928 cpu_manager.go:414] "containerMap: removing container" podUID="04c637e6-143b-4813-b498-49e8cded4614" containerName="maas-api"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401634    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c637e6-143b-4813-b498-49e8cded4614" containerName="maas-api"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401685    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401694    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401705    4928 cpu_manager.go:414] "containerMap: removing container" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401712    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401724    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401731    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401923    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401940    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401952    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcab597-c6c0-4577-b9fa-a1e20c567b56" containerName="payload-pre-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401965    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e3bcb6-f16b-4b8d-a9f0-6c5fa7be54d5" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401973    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8cd4fc-9962-49f3-a611-150cc57dac57" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401983    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.401995    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95d9c1b-9742-4173-add0-df61d21600a9" containerName="maas-api"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.402006    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3262cb-20d2-473c-9e48-49bc20b0b560" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.402015    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5ddc38-0f47-4406-b626-cf38b37cdd43" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.402029    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f79307-ad18-4efa-bfe4-13d23e590c56" containerName="payload-processing"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.402039    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="398d8f4d-80fe-4e36-b701-2468d193bac8" containerName="istio-proxy"
Nov 02 13:22:43.418174 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.402050    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c637e6-143b-4813-b498-49e8cded4614" containerName="maas-api"
Nov 02 13:22:43.421376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.420946    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.441935 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.433869    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.442734    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.442763    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.442794    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.442802    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.443033    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e41d50-b9be-47bb-a01f-5f0b631596bb" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.443047    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6dfe07-9efb-4437-a111-d30aabccf018" containerName="istio-proxy"
Nov 02 13:22:43.445281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.443257    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5m42\" (UniqueName: \"kubernetes.io/projected/d95d9c1b-9742-4173-add0-df61d21600a9-kube-api-access-f5m42\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:43.445963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.445933    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs" pod "d95d9c1b-9742-4173-add0-df61d21600a9" (UID: "d95d9c1b-9742-4173-add0-df61d21600a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:43.449888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.449782    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:22:43.450023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.449943    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.450109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.450087    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:22:43.544759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.544653    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.544759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.544712    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.544796    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.544932    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.544973    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545063    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545132    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545182    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545308    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545361    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6l8m\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545455    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545604 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545581    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545634    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545686    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545744    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545797    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.545976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545876    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.545976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.545920    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvkw\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.546183 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.546083    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95d9c1b-9742-4173-add0-df61d21600a9-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:43.647333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647293    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647340    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvkw\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647402    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647426    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.647530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647449    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647531    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647570    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647608    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647655    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647707    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.647779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647777    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647818    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6l8m\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647890    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647898    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.647932    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648021    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648050    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648100    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648163    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648181    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648198    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648199 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648194    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648585    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.648801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648608    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648678    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.648997 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.648893    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.649249 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.649226    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.649920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.649896    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.650773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.650751    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.651034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.651015    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.651129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.651061    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.651271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.651251    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.673521 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.673469    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.673619 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.673519    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6l8m\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2t5gf\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.673754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.673731    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvkw\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.673798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.673737    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rhhl4\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.764482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.764438    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:43.779708 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.779676    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:43.815968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.815180    4928 generic.go:356] "Generic (PLEG): container finished" podID="d95d9c1b-9742-4173-add0-df61d21600a9" containerID="aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5" exitCode=0
Nov 02 13:22:43.815968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.815331    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" event={"ID":"d95d9c1b-9742-4173-add0-df61d21600a9","Type":"ContainerDied","Data":"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"}
Nov 02 13:22:43.815968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.815384    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq" event={"ID":"d95d9c1b-9742-4173-add0-df61d21600a9","Type":"ContainerDied","Data":"d4592679b8af85ade8da97d9a289dfc369720dc9de6e973981efbdd2806dbb2d"}
Nov 02 13:22:43.815968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.815419    4928 scope.go:122] "RemoveContainer" containerID="aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"
Nov 02 13:22:43.815968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.815574    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"
Nov 02 13:22:43.848230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.848078    4928 scope.go:122] "RemoveContainer" containerID="aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"
Nov 02 13:22:43.848864 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:43.848439    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5\": container with ID starting with aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5 not found: ID does not exist" containerID="aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"
Nov 02 13:22:43.848864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.848478    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5"} err="failed to get container status \"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5\": rpc error: code = NotFound desc = could not find container \"aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5\": container with ID starting with aaa3d0e4cbeca741484b5c2ccca7fa2023ad394b8c4f4c23cda6def196adf1c5 not found: ID does not exist"
Nov 02 13:22:43.903902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.903870    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:22:43.914972 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:43.914943    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361c3a-58869477bb-4q2fq"]
Nov 02 13:22:44.113974 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:22:44.113941    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3f42a5_3bcd_4e40_aa0a_ef7a67b53f33.slice/crio-cc2e129c75fc67783cb91ae14f08d3485f027ffd9da86c754aaf31379ae28c11 WatchSource:0}: Error finding container cc2e129c75fc67783cb91ae14f08d3485f027ffd9da86c754aaf31379ae28c11: Status 404 returned error can't find the container with id cc2e129c75fc67783cb91ae14f08d3485f027ffd9da86c754aaf31379ae28c11
Nov 02 13:22:44.114319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.114285    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:22:44.117986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.117919    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.118112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.118028    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.118112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.118067    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.120284 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:22:44.120245    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff606135_ca03_429b_9023_69a7976bee38.slice/crio-3c8d2bbcae7784de062ae581617930fda1a2a917075131078a48865709be108c WatchSource:0}: Error finding container 3c8d2bbcae7784de062ae581617930fda1a2a917075131078a48865709be108c: Status 404 returned error can't find the container with id 3c8d2bbcae7784de062ae581617930fda1a2a917075131078a48865709be108c
Nov 02 13:22:44.123676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.123650    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:22:44.124322 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.124285    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.124409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.124360    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.124409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.124402    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:22:44.502532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.502487    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95d9c1b-9742-4173-add0-df61d21600a9" path="/var/lib/kubelet/pods/d95d9c1b-9742-4173-add0-df61d21600a9/volumes"
Nov 02 13:22:44.579041 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.578990    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" podUID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" containerName="payload-processing" containerID="cri-o://e1de749018a221ced99db7083682c13627273b876c559e57779c116b9fee131c" gracePeriod=25
Nov 02 13:22:44.824782 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.824738    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" event={"ID":"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33","Type":"ContainerStarted","Data":"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"}
Nov 02 13:22:44.825744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.824811    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" event={"ID":"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33","Type":"ContainerStarted","Data":"cc2e129c75fc67783cb91ae14f08d3485f027ffd9da86c754aaf31379ae28c11"}
Nov 02 13:22:44.835713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.835679    4928 generic.go:356] "Generic (PLEG): container finished" podID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" containerID="e1de749018a221ced99db7083682c13627273b876c559e57779c116b9fee131c" exitCode=0
Nov 02 13:22:44.835860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.835752    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" event={"ID":"0ae833f6-01fa-4f9f-b85e-44730039ae2e","Type":"ContainerDied","Data":"e1de749018a221ced99db7083682c13627273b876c559e57779c116b9fee131c"}
Nov 02 13:22:44.838001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.837969    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" event={"ID":"ff606135-ca03-429b-9023-69a7976bee38","Type":"ContainerStarted","Data":"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"}
Nov 02 13:22:44.838150 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.838022    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" event={"ID":"ff606135-ca03-429b-9023-69a7976bee38","Type":"ContainerStarted","Data":"3c8d2bbcae7784de062ae581617930fda1a2a917075131078a48865709be108c"}
Nov 02 13:22:44.878744 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.878687    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podStartSLOduration=1.8786720620000001 podStartE2EDuration="1.878672062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:44.873776582 +0000 UTC m=+3362.522830552" watchObservedRunningTime="2026-11-02 13:22:44.878672062 +0000 UTC m=+3362.527726019"
Nov 02 13:22:44.880542 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.880507    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:22:44.928601 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.928546    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podStartSLOduration=1.928531317 podStartE2EDuration="1.928531317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:44.922195012 +0000 UTC m=+3362.571248970" watchObservedRunningTime="2026-11-02 13:22:44.928531317 +0000 UTC m=+3362.577585274"
Nov 02 13:22:44.975544 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.974030    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\" (UniqueName: \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\") pod \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") "
Nov 02 13:22:44.975544 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.974167    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\") pod \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\" (UID: \"0ae833f6-01fa-4f9f-b85e-44730039ae2e\") "
Nov 02 13:22:44.976537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.976266    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume" pod "0ae833f6-01fa-4f9f-b85e-44730039ae2e" (UID: "0ae833f6-01fa-4f9f-b85e-44730039ae2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:44.985877 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:44.977813    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds" pod "0ae833f6-01fa-4f9f-b85e-44730039ae2e" (UID: "0ae833f6-01fa-4f9f-b85e-44730039ae2e"). InnerVolumeSpecName "kube-api-access-jr8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:45.076850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.076744    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8ds\" (UniqueName: \"kubernetes.io/projected/0ae833f6-01fa-4f9f-b85e-44730039ae2e-kube-api-access-jr8ds\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:45.076850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.076788    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae833f6-01fa-4f9f-b85e-44730039ae2e-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:45.770853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.765985    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:45.770853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.767937    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:45.770853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.768010    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:45.782857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.781941    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:45.784856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.783520    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:45.784856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.783595    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:45.893228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.893186    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"
Nov 02 13:22:45.893228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.893214    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t" event={"ID":"0ae833f6-01fa-4f9f-b85e-44730039ae2e","Type":"ContainerDied","Data":"5e8fe30a3718ec5e814659e70201952005a0771c584a5b090765fcb29ea84974"}
Nov 02 13:22:45.893815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.893288    4928 scope.go:122] "RemoveContainer" containerID="e1de749018a221ced99db7083682c13627273b876c559e57779c116b9fee131c"
Nov 02 13:22:45.980852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.974082    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:22:45.993110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:45.993065    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-361c3a-5f8f84cb7c-p6c7t"]
Nov 02 13:22:46.501538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:46.501500    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" path="/var/lib/kubelet/pods/0ae833f6-01fa-4f9f-b85e-44730039ae2e/volumes"
Nov 02 13:22:46.768906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:46.768240    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:46.768906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:46.768302    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:46.787382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:46.784909    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:46.787382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:46.784972    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:47.772851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:47.771908    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:47.772851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:47.771968    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:47.787654 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:47.787612    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:47.787820 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:47.787691    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:48.212932 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.212464    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:22:48.212932 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.212763    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" containerID="cri-o://1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1" gracePeriod=30
Nov 02 13:22:48.774537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.774497    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:48.775166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.775132    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:48.782382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.782231    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:48.782382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:48.782298    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:49.778853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:49.765269    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:49.778853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:49.765337    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:49.780857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:49.780298    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:49.780857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:49.780367    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:50.765080 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:50.765046    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:50.765252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:50.765116    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:50.780574 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:50.780541    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:50.780975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:50.780593    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:51.735320 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.735275    4928 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:51.735652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.735350    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:51.765374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.765332    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:51.765586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.765402    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:51.780493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.780459    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:22:51.780650 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:51.780521    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 13:22:52.765535 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:52.765497    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 13:22:52.765760 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:52.765553    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 13:22:53.483337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.483297    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:53.484765 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.484737    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:53.485467 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.485443    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:22:53.807231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.807136    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:53.807820 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.807753    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:53.809026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.808990    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:22:53.879115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:53.879087    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:22:54.035531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035446    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035539    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035561    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035686    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035948 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035748    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:54.035948 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035796    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035948 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035900    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.035948 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035922    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:54.036160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.035965    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.036160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036061    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.036160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036118    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\") pod \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\" (UID: \"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab\") "
Nov 02 13:22:54.036160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036142    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:54.036379 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036335    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:54.036532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036497    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:54.036979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036951    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.037088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.036984    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.037088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.037002    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.037088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.037018    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.037088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.037033    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.038144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.038116    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:54.038268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.038204    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:54.038268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.038220    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:54.038354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.038296    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw" pod "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" (UID: "a53378b8-f9ab-4c5a-83dc-f6bed24b7aab"). InnerVolumeSpecName "kube-api-access-ts4fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:54.112301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.112244    4928 generic.go:356] "Generic (PLEG): container finished" podID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerID="1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1" exitCode=0
Nov 02 13:22:54.113286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.113255    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"
Nov 02 13:22:54.114448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.113983    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerDied","Data":"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"}
Nov 02 13:22:54.114448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.114045    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs" event={"ID":"a53378b8-f9ab-4c5a-83dc-f6bed24b7aab","Type":"ContainerDied","Data":"4e7dc54e0f2329687ac4f39f7b7a21c719f04002a66a932e5178c51aa13c0c55"}
Nov 02 13:22:54.114448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.114074    4928 scope.go:122] "RemoveContainer" containerID="1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"
Nov 02 13:22:54.144482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.144452    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.144640 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.144486    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.144640 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.144500    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.144640 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.144514    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4fw\" (UniqueName: \"kubernetes.io/projected/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab-kube-api-access-ts4fw\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:22:54.146975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.146954    4928 scope.go:122] "RemoveContainer" containerID="d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"
Nov 02 13:22:54.157635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.157618    4928 scope.go:122] "RemoveContainer" containerID="1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"
Nov 02 13:22:54.157950 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:54.157926    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1\": container with ID starting with 1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1 not found: ID does not exist" containerID="1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"
Nov 02 13:22:54.158038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.157958    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1"} err="failed to get container status \"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1\": rpc error: code = NotFound desc = could not find container \"1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1\": container with ID starting with 1e2beb576ae51b49ade08a61b49281dafc0a9fcfa99b406908b96e14eaf980e1 not found: ID does not exist"
Nov 02 13:22:54.158038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.157975    4928 scope.go:122] "RemoveContainer" containerID="d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"
Nov 02 13:22:54.158294 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:22:54.158268    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc\": container with ID starting with d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc not found: ID does not exist" containerID="d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"
Nov 02 13:22:54.158396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.158296    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc"} err="failed to get container status \"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc\": rpc error: code = NotFound desc = could not find container \"d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc\": container with ID starting with d48214352cc87eb4d9bfe973dd6e1e2d944e8f6f7d83a56895faf480767db9dc not found: ID does not exist"
Nov 02 13:22:54.173960 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.173925    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:22:54.181163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.181136    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-g86hs"]
Nov 02 13:22:54.499440 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:22:54.499402    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" path="/var/lib/kubelet/pods/a53378b8-f9ab-4c5a-83dc-f6bed24b7aab/volumes"
Nov 02 13:23:02.767227 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:02.765958    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:23:02.767227 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:02.766217    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" containerID="cri-o://34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2" gracePeriod=30
Nov 02 13:23:05.904455 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:05.904407    4928 patch_prober.go:29] interesting pod/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:23:05.904947 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:05.904472    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:23:08.057656 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.057588    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:23:08.235573 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.235537    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.235780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.235761    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.235902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.235843    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.235975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.235941    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.235986    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236092    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236119    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:08.236167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236146    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236274 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236185    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236274 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236236    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\") pod \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\" (UID: \"af290eea-132b-4d9a-83f0-1edbc5aaca2a\") "
Nov 02 13:23:08.236647 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236620    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:08.236647 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236620    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:08.236781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236694    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:08.236910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236891    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:08.237001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236893    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.237001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236933    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.237001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236950    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.237001 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.236958    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.238969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.238885    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:23:08.239123 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.239099    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:08.240007 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.239976    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "kube-api-access-7r5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:08.240129 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.240019    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token" pod "af290eea-132b-4d9a-83f0-1edbc5aaca2a" (UID: "af290eea-132b-4d9a-83f0-1edbc5aaca2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:08.248677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.248645    4928 generic.go:356] "Generic (PLEG): container finished" podID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerID="34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2" exitCode=0
Nov 02 13:23:08.248786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.248695    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"
Nov 02 13:23:08.248786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.248718    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerDied","Data":"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"}
Nov 02 13:23:08.248786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.248774    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh" event={"ID":"af290eea-132b-4d9a-83f0-1edbc5aaca2a","Type":"ContainerDied","Data":"02ba20ba6c1c2c7055da24ce090c2d7ff82fe74f13d2a3d37af12553bc1148ab"}
Nov 02 13:23:08.248994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.248805    4928 scope.go:122] "RemoveContainer" containerID="34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"
Nov 02 13:23:08.274372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.274349    4928 scope.go:122] "RemoveContainer" containerID="901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"
Nov 02 13:23:08.285690 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.285671    4928 scope.go:122] "RemoveContainer" containerID="34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"
Nov 02 13:23:08.286004 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:23:08.285980    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2\": container with ID starting with 34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2 not found: ID does not exist" containerID="34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"
Nov 02 13:23:08.286114 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.286010    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2"} err="failed to get container status \"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2\": rpc error: code = NotFound desc = could not find container \"34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2\": container with ID starting with 34b95c864c92f6bdd4d3cf26d2bd782043697040845467aaab0529ea374725a2 not found: ID does not exist"
Nov 02 13:23:08.286114 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.286031    4928 scope.go:122] "RemoveContainer" containerID="901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"
Nov 02 13:23:08.286293 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:23:08.286274    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b\": container with ID starting with 901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b not found: ID does not exist" containerID="901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"
Nov 02 13:23:08.286364 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.286297    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b"} err="failed to get container status \"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b\": rpc error: code = NotFound desc = could not find container \"901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b\": container with ID starting with 901f2f70000156ecf2b97092db0b5e9ab271dad376a42f9a03af5c257a70c60b not found: ID does not exist"
Nov 02 13:23:08.312478 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.312383    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:23:08.316603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.316576    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-48d74e-openshift-default-5b465bf649-x9fwh"]
Nov 02 13:23:08.338473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.338432    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.338473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.338469    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5vk\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-kube-api-access-7r5vk\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.338473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.338479    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.338743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.338489    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.338743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.338500    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af290eea-132b-4d9a-83f0-1edbc5aaca2a-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:23:08.499769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:23:08.499732    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" path="/var/lib/kubelet/pods/af290eea-132b-4d9a-83f0-1edbc5aaca2a/volumes"
Nov 02 13:23:09.730329 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:23:09.730291    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:23:39.730778 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:23:39.730737    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:24:09.730939 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:24:09.730895    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:24:39.732483 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:24:39.732440    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:25:09.733045 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:25:09.733004    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:25:24.542957 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:24.542915    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rhhl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:25:24.543801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:24.543512    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:25:24.860055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:24.859953    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2t5gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:25:24.860341 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:24.860306    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:25:39.731048 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:25:39.730998    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:25:47.970637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:47.970598    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:25:47.971178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:47.970794    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy" containerID="cri-o://015e75f7718de83aab5004221260152d3f81c18d4bb0d9531dcee0339aa16706" gracePeriod=30
Nov 02 13:25:53.192520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.192477    4928 generic.go:356] "Generic (PLEG): container finished" podID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerID="015e75f7718de83aab5004221260152d3f81c18d4bb0d9531dcee0339aa16706" exitCode=0
Nov 02 13:25:53.193056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.192927    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerDied","Data":"015e75f7718de83aab5004221260152d3f81c18d4bb0d9531dcee0339aa16706"}
Nov 02 13:25:53.193208 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.193124    4928 scope.go:122] "RemoveContainer" containerID="9a9a831118192fd865132e9f83f48e3fef88eda98daa2f5f41b269cc3e2e647c"
Nov 02 13:25:53.291200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.291172    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:25:53.407652 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.407608    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.407873 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.407661    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.407961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.407820    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.407961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.407941    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:53.408088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408010    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.408149 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408104    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.408206 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408161    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.408302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408277    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:53.408366 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408311    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.408366 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408352    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.408479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408443    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\") pod \"107a0384-09e7-42d3-902e-3f506d80c2fb\" (UID: \"107a0384-09e7-42d3-902e-3f506d80c2fb\") "
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408578    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.408657    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409003    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409215    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409239    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409252    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409266    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107a0384-09e7-42d3-902e-3f506d80c2fb-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.409348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.409280    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.410456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.410433    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:53.410549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.410497    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:25:53.410678 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.410657    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:53.410946 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.410927    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht" pod "107a0384-09e7-42d3-902e-3f506d80c2fb" (UID: "107a0384-09e7-42d3-902e-3f506d80c2fb"). InnerVolumeSpecName "kube-api-access-d7bht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:53.511372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.511288    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.511372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.511320    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7bht\" (UniqueName: \"kubernetes.io/projected/107a0384-09e7-42d3-902e-3f506d80c2fb-kube-api-access-d7bht\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.511372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.511332    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:53.511372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:53.511343    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107a0384-09e7-42d3-902e-3f506d80c2fb-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:25:54.203468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.203439    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"
Nov 02 13:25:54.203468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.203451    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7" event={"ID":"107a0384-09e7-42d3-902e-3f506d80c2fb","Type":"ContainerDied","Data":"6db7f652ed38544ff805ada6f648a00395daf1b67d59f48b0b5c73b76e313b17"}
Nov 02 13:25:54.204086 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.203522    4928 scope.go:122] "RemoveContainer" containerID="015e75f7718de83aab5004221260152d3f81c18d4bb0d9531dcee0339aa16706"
Nov 02 13:25:54.277949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.277914    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:25:54.291493 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.291462    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-mpjw7"]
Nov 02 13:25:54.498717 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:25:54.498682    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" path="/var/lib/kubelet/pods/107a0384-09e7-42d3-902e-3f506d80c2fb/volumes"
Nov 02 13:26:09.732615 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:26:09.732572    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:26:34.354151 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:34.354115    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:26:34.355002 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:34.354340    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" podUID="797f5096-ae6a-4454-a351-25751edae1af" containerName="maas-api" containerID="cri-o://99d320a444068816e0147c8a8a33987a39c1e8419845abb12fdc188cf3ac0530" gracePeriod=30
Nov 02 13:26:34.511895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:34.511799    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" podUID="797f5096-ae6a-4454-a351-25751edae1af" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.214:8443/health\": dial tcp 10.132.0.214:8443: connect: connection refused"
Nov 02 13:26:34.721697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:34.720777    4928 generic.go:356] "Generic (PLEG): container finished" podID="797f5096-ae6a-4454-a351-25751edae1af" containerID="99d320a444068816e0147c8a8a33987a39c1e8419845abb12fdc188cf3ac0530" exitCode=0
Nov 02 13:26:34.721697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:34.720855    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" event={"ID":"797f5096-ae6a-4454-a351-25751edae1af","Type":"ContainerDied","Data":"99d320a444068816e0147c8a8a33987a39c1e8419845abb12fdc188cf3ac0530"}
Nov 02 13:26:35.124321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.124290    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:26:35.257371 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.257312    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") pod \"797f5096-ae6a-4454-a351-25751edae1af\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") "
Nov 02 13:26:35.257567 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.257452    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\") pod \"797f5096-ae6a-4454-a351-25751edae1af\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") "
Nov 02 13:26:35.257567 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.257523    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\") pod \"797f5096-ae6a-4454-a351-25751edae1af\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") "
Nov 02 13:26:35.257731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.257578    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\" (UniqueName: \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\") pod \"797f5096-ae6a-4454-a351-25751edae1af\" (UID: \"797f5096-ae6a-4454-a351-25751edae1af\") "
Nov 02 13:26:35.263436 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.263400    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca" pod "797f5096-ae6a-4454-a351-25751edae1af" (UID: "797f5096-ae6a-4454-a351-25751edae1af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:35.264598 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.264572    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs" pod "797f5096-ae6a-4454-a351-25751edae1af" (UID: "797f5096-ae6a-4454-a351-25751edae1af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:35.266489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.266465    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls" pod "797f5096-ae6a-4454-a351-25751edae1af" (UID: "797f5096-ae6a-4454-a351-25751edae1af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:35.270177 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.270152    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7" pod "797f5096-ae6a-4454-a351-25751edae1af" (UID: "797f5096-ae6a-4454-a351-25751edae1af"). InnerVolumeSpecName "kube-api-access-g2lg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:35.359614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.359577    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797f5096-ae6a-4454-a351-25751edae1af-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:35.359614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.359608    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2lg7\" (UniqueName: \"kubernetes.io/projected/797f5096-ae6a-4454-a351-25751edae1af-kube-api-access-g2lg7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:35.359614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.359621    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:35.360094 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.359631    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797f5096-ae6a-4454-a351-25751edae1af-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:35.731416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.731322    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms" event={"ID":"797f5096-ae6a-4454-a351-25751edae1af","Type":"ContainerDied","Data":"8b9bd15b4e60d1ee4d8b6737c13c033810a4ce8735018dd7045665e3756c5d4a"}
Nov 02 13:26:35.731416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.731350    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"
Nov 02 13:26:35.731416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.731394    4928 scope.go:122] "RemoveContainer" containerID="99d320a444068816e0147c8a8a33987a39c1e8419845abb12fdc188cf3ac0530"
Nov 02 13:26:35.794693 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.794659    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:26:35.807976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:35.807950    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e8a402-5b89889cf4-6n2ms"]
Nov 02 13:26:36.500231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:36.500197    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797f5096-ae6a-4454-a351-25751edae1af" path="/var/lib/kubelet/pods/797f5096-ae6a-4454-a351-25751edae1af/volumes"
Nov 02 13:26:39.730315 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:26:39.730272    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:26:41.671711 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:41.671674    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:26:41.672326 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:41.671951    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" containerID="cri-o://d95a5913f326b9898622df240e7cac18d5ad0350ce1dc8066f9217e74041fd97" gracePeriod=30
Nov 02 13:26:43.304036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:43.304001    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:26:43.317287 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:43.317256    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:26:44.699017 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:44.698968    4928 patch_prober.go:29] interesting pod/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:26:44.699485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:44.699056    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:26:46.891580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:46.891534    4928 generic.go:356] "Generic (PLEG): container finished" podID="7344f928-7931-47b9-b15a-58fc25d6e878" containerID="d95a5913f326b9898622df240e7cac18d5ad0350ce1dc8066f9217e74041fd97" exitCode=0
Nov 02 13:26:46.892083 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:46.891743    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerDied","Data":"d95a5913f326b9898622df240e7cac18d5ad0350ce1dc8066f9217e74041fd97"}
Nov 02 13:26:46.892083 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:46.891802    4928 scope.go:122] "RemoveContainer" containerID="151c63225a24b8706d6435cc186c4f9960566a28f60ff1120f55c56a2f7156bc"
Nov 02 13:26:46.981810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:46.981782    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:26:47.079381 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079346    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.079596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079425    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.079596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079583    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.079740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079640    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.079740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079696    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.079892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079750    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.080318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080131    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:47.080318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080262    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:47.080318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080229    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:47.080318 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.079863    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.080661 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080423    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.080661 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080472    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\") pod \"7344f928-7931-47b9-b15a-58fc25d6e878\" (UID: \"7344f928-7931-47b9-b15a-58fc25d6e878\") "
Nov 02 13:26:47.080783 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080704    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:47.080783 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.080728    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:47.081380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.081266    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344f928-7931-47b9-b15a-58fc25d6e878-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.081380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.081294    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.081380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.081308    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.081380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.081321    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.081380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.081335    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.082406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.082366    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:47.082533 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.082422    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "kube-api-access-9mrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:47.083070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.083037    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:47.083180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.083076    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo" pod "7344f928-7931-47b9-b15a-58fc25d6e878" (UID: "7344f928-7931-47b9-b15a-58fc25d6e878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:26:47.182889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.182845    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.182889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.182876    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344f928-7931-47b9-b15a-58fc25d6e878-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.182889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.182893    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrpx\" (UniqueName: \"kubernetes.io/projected/7344f928-7931-47b9-b15a-58fc25d6e878-kube-api-access-9mrpx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.183145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.182908    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344f928-7931-47b9-b15a-58fc25d6e878-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:26:47.900529 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.900493    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"
Nov 02 13:26:47.900529 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.900522    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc" event={"ID":"7344f928-7931-47b9-b15a-58fc25d6e878","Type":"ContainerDied","Data":"2af26a364777926370d77c655752e5bba41ce903a5494e656327313443573333"}
Nov 02 13:26:47.901046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.900611    4928 scope.go:122] "RemoveContainer" containerID="d95a5913f326b9898622df240e7cac18d5ad0350ce1dc8066f9217e74041fd97"
Nov 02 13:26:47.978873 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.978811    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:26:47.982763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:47.982739    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e8a402-openshift-default-869f646c59-qglhc"]
Nov 02 13:26:48.499563 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:26:48.499526    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" path="/var/lib/kubelet/pods/7344f928-7931-47b9-b15a-58fc25d6e878/volumes"
Nov 02 13:27:09.731266 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:27:09.731221    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:27:30.205998 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.205906    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:27:30.206486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.206143    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" containerName="maas-api" containerID="cri-o://aef28d120bd5b941c75294fd9a7dc9c3e1c192df261311da4a1fa9dc31e4f982" gracePeriod=30
Nov 02 13:27:30.338513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.338473    4928 generic.go:356] "Generic (PLEG): container finished" podID="bfa0c403-890b-4006-ad32-eac4cb160996" containerID="aef28d120bd5b941c75294fd9a7dc9c3e1c192df261311da4a1fa9dc31e4f982" exitCode=0
Nov 02 13:27:30.338702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.338546    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" event={"ID":"bfa0c403-890b-4006-ad32-eac4cb160996","Type":"ContainerDied","Data":"aef28d120bd5b941c75294fd9a7dc9c3e1c192df261311da4a1fa9dc31e4f982"}
Nov 02 13:27:30.379620 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.379573    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.217:8443/health\": dial tcp 10.132.0.217:8443: connect: connection refused"
Nov 02 13:27:30.705981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.705950    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:27:30.823408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.823362    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\") pod \"bfa0c403-890b-4006-ad32-eac4cb160996\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") "
Nov 02 13:27:30.823589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.823481    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\" (UniqueName: \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\") pod \"bfa0c403-890b-4006-ad32-eac4cb160996\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") "
Nov 02 13:27:30.823589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.823540    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\") pod \"bfa0c403-890b-4006-ad32-eac4cb160996\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") "
Nov 02 13:27:30.823755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.823726    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\") pod \"bfa0c403-890b-4006-ad32-eac4cb160996\" (UID: \"bfa0c403-890b-4006-ad32-eac4cb160996\") "
Nov 02 13:27:30.824145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.824115    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca" pod "bfa0c403-890b-4006-ad32-eac4cb160996" (UID: "bfa0c403-890b-4006-ad32-eac4cb160996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:30.824443 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.824417    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa0c403-890b-4006-ad32-eac4cb160996-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:30.826679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.826632    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs" pod "bfa0c403-890b-4006-ad32-eac4cb160996" (UID: "bfa0c403-890b-4006-ad32-eac4cb160996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:30.829361 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.829330    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls" pod "bfa0c403-890b-4006-ad32-eac4cb160996" (UID: "bfa0c403-890b-4006-ad32-eac4cb160996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:30.829891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.829865    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95" pod "bfa0c403-890b-4006-ad32-eac4cb160996" (UID: "bfa0c403-890b-4006-ad32-eac4cb160996"). InnerVolumeSpecName "kube-api-access-2fz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:30.926321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.926281    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:30.926321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.926324    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fz95\" (UniqueName: \"kubernetes.io/projected/bfa0c403-890b-4006-ad32-eac4cb160996-kube-api-access-2fz95\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:30.926527 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:30.926342    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa0c403-890b-4006-ad32-eac4cb160996-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:31.350163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:31.350132    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"
Nov 02 13:27:31.350614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:31.350129    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb" event={"ID":"bfa0c403-890b-4006-ad32-eac4cb160996","Type":"ContainerDied","Data":"c9d18e292cf058938ad14e365fd5eac13e2f9cb7fd269286dc6ac97be7e46e56"}
Nov 02 13:27:31.350614 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:31.350283    4928 scope.go:122] "RemoveContainer" containerID="aef28d120bd5b941c75294fd9a7dc9c3e1c192df261311da4a1fa9dc31e4f982"
Nov 02 13:27:31.422119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:31.422093    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:27:31.432553 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:31.432522    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-357b30-6bb4ff55d9-lr4kb"]
Nov 02 13:27:32.499666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:32.499631    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" path="/var/lib/kubelet/pods/bfa0c403-890b-4006-ad32-eac4cb160996/volumes"
Nov 02 13:27:35.625066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:35.625017    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:27:35.625586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:35.625283    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy" containerID="cri-o://877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9" gracePeriod=30
Nov 02 13:27:39.730324 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:27:39.730281    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:27:40.924637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:40.924612    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:27:41.100427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100389    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100531    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100607    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100696 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100655    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100693    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100726    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.100860 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100767    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:41.100942 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100915    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.101021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.100975    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:41.101021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101005    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.101146 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101049    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\") pod \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\" (UID: \"4bef96c2-d223-4cd4-bc4f-8df0a100184b\") "
Nov 02 13:27:41.101146 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101073    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:41.101454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101376    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:41.101454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101383    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:41.101891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101750    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.101891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101780    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.101891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101798    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.101891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101812    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.101891 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.101866    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.103118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.103082    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:41.103359 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.103334    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "kube-api-access-qxgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:41.103427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.103353    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:41.103629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.103608    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo" pod "4bef96c2-d223-4cd4-bc4f-8df0a100184b" (UID: "4bef96c2-d223-4cd4-bc4f-8df0a100184b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:27:41.203745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.203657    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.203745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.203687    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.203745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.203696    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgvf\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-kube-api-access-qxgvf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.203745 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.203706    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bef96c2-d223-4cd4-bc4f-8df0a100184b-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:41.439999 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.439966    4928 generic.go:356] "Generic (PLEG): container finished" podID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerID="877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9" exitCode=0
Nov 02 13:27:41.440198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.440030    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"
Nov 02 13:27:41.440198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.440051    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerDied","Data":"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"}
Nov 02 13:27:41.440198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.440135    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j" event={"ID":"4bef96c2-d223-4cd4-bc4f-8df0a100184b","Type":"ContainerDied","Data":"e6d3be34d665ef3d784d1c11edef3251534b351864e7e61506aa9b9cc3163221"}
Nov 02 13:27:41.440198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.440163    4928 scope.go:122] "RemoveContainer" containerID="877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"
Nov 02 13:27:41.452148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.452126    4928 scope.go:122] "RemoveContainer" containerID="168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"
Nov 02 13:27:41.462516 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.462489    4928 scope.go:122] "RemoveContainer" containerID="877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"
Nov 02 13:27:41.462789 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:27:41.462772    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9\": container with ID starting with 877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9 not found: ID does not exist" containerID="877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"
Nov 02 13:27:41.462901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.462799    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9"} err="failed to get container status \"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9\": rpc error: code = NotFound desc = could not find container \"877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9\": container with ID starting with 877c92f15ad398baa5d5e77858337b61abad4c6d9657ac5a4ac1de80f1e020c9 not found: ID does not exist"
Nov 02 13:27:41.462901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.462815    4928 scope.go:122] "RemoveContainer" containerID="168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"
Nov 02 13:27:41.463080 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:27:41.463065    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108\": container with ID starting with 168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108 not found: ID does not exist" containerID="168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"
Nov 02 13:27:41.463139 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.463082    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108"} err="failed to get container status \"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108\": rpc error: code = NotFound desc = could not find container \"168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108\": container with ID starting with 168e1c5f7a92ce91f90643e9b54847a0d006511b7cf1836b744a9f90befca108 not found: ID does not exist"
Nov 02 13:27:41.514719 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.514688    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:27:41.525863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:41.525817    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-357b30-openshift-default-85bd97c49-nvq5j"]
Nov 02 13:27:42.498819 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:42.498786    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" path="/var/lib/kubelet/pods/4bef96c2-d223-4cd4-bc4f-8df0a100184b/volumes"
Nov 02 13:27:58.747768 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:58.747728    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:27:58.748458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:58.748421    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" podUID="8b646e44-487a-4ea8-a520-6c7a12f772b4" containerName="maas-api" containerID="cri-o://b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67" gracePeriod=30
Nov 02 13:27:59.094315 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.094266    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:27:59.255210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.254248    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\" (UniqueName: \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\") pod \"8b646e44-487a-4ea8-a520-6c7a12f772b4\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") "
Nov 02 13:27:59.255210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.254339    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\") pod \"8b646e44-487a-4ea8-a520-6c7a12f772b4\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") "
Nov 02 13:27:59.255210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.254379    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\") pod \"8b646e44-487a-4ea8-a520-6c7a12f772b4\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") "
Nov 02 13:27:59.255210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.254576    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\") pod \"8b646e44-487a-4ea8-a520-6c7a12f772b4\" (UID: \"8b646e44-487a-4ea8-a520-6c7a12f772b4\") "
Nov 02 13:27:59.255600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.255578    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca" pod "8b646e44-487a-4ea8-a520-6c7a12f772b4" (UID: "8b646e44-487a-4ea8-a520-6c7a12f772b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:59.264847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.259012    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls" pod "8b646e44-487a-4ea8-a520-6c7a12f772b4" (UID: "8b646e44-487a-4ea8-a520-6c7a12f772b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:59.264847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.259093    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs" pod "8b646e44-487a-4ea8-a520-6c7a12f772b4" (UID: "8b646e44-487a-4ea8-a520-6c7a12f772b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:59.264847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.261343    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4" pod "8b646e44-487a-4ea8-a520-6c7a12f772b4" (UID: "8b646e44-487a-4ea8-a520-6c7a12f772b4"). InnerVolumeSpecName "kube-api-access-fg6c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:59.361112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.361076    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b646e44-487a-4ea8-a520-6c7a12f772b4-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:59.361112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.361113    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6c4\" (UniqueName: \"kubernetes.io/projected/8b646e44-487a-4ea8-a520-6c7a12f772b4-kube-api-access-fg6c4\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:59.361417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.361133    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:59.361417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.361145    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b646e44-487a-4ea8-a520-6c7a12f772b4-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:27:59.595032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.594939    4928 generic.go:356] "Generic (PLEG): container finished" podID="8b646e44-487a-4ea8-a520-6c7a12f772b4" containerID="b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67" exitCode=0
Nov 02 13:27:59.595032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.594980    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" event={"ID":"8b646e44-487a-4ea8-a520-6c7a12f772b4","Type":"ContainerDied","Data":"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"}
Nov 02 13:27:59.595032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.595004    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"
Nov 02 13:27:59.595032 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.595032    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj" event={"ID":"8b646e44-487a-4ea8-a520-6c7a12f772b4","Type":"ContainerDied","Data":"f3b5d359e91ccc9678f6bf4def1987caf394eb6c7589cc3a708c8dd3d0659848"}
Nov 02 13:27:59.595355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.595054    4928 scope.go:122] "RemoveContainer" containerID="b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"
Nov 02 13:27:59.607662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.607584    4928 scope.go:122] "RemoveContainer" containerID="b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"
Nov 02 13:27:59.651340 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:27:59.651297    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67\": container with ID starting with b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67 not found: ID does not exist" containerID="b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"
Nov 02 13:27:59.651522 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.651349    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67"} err="failed to get container status \"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67\": rpc error: code = NotFound desc = could not find container \"b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67\": container with ID starting with b8af1d72723ad332189be903f85a146c9dfdf131c0fe43b22befab9855302d67 not found: ID does not exist"
Nov 02 13:27:59.678292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.678261    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:27:59.690864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:27:59.690817    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8962e9-576c5769d-qrvpj"]
Nov 02 13:28:00.499917 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:00.499874    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b646e44-487a-4ea8-a520-6c7a12f772b4" path="/var/lib/kubelet/pods/8b646e44-487a-4ea8-a520-6c7a12f772b4/volumes"
Nov 02 13:28:09.731949 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:28:09.731910    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:28:39.729697 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:28:39.729659    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:28:39.992671 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:39.992631    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:28:39.992962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:39.992927    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" podUID="87a0322b-c09b-43ef-974a-771a7a66fc47" containerName="maas-api" containerID="cri-o://b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3" gracePeriod=30
Nov 02 13:28:40.474848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.474309    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:28:40.572101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.570303    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\") pod \"87a0322b-c09b-43ef-974a-771a7a66fc47\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") "
Nov 02 13:28:40.572101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.570389    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\") pod \"87a0322b-c09b-43ef-974a-771a7a66fc47\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") "
Nov 02 13:28:40.572101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.570471    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\") pod \"87a0322b-c09b-43ef-974a-771a7a66fc47\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") "
Nov 02 13:28:40.572101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.570524    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\" (UniqueName: \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\") pod \"87a0322b-c09b-43ef-974a-771a7a66fc47\" (UID: \"87a0322b-c09b-43ef-974a-771a7a66fc47\") "
Nov 02 13:28:40.572101 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.571993    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca" pod "87a0322b-c09b-43ef-974a-771a7a66fc47" (UID: "87a0322b-c09b-43ef-974a-771a7a66fc47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:40.573743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.573705    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs" pod "87a0322b-c09b-43ef-974a-771a7a66fc47" (UID: "87a0322b-c09b-43ef-974a-771a7a66fc47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:28:40.575205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.575181    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls" pod "87a0322b-c09b-43ef-974a-771a7a66fc47" (UID: "87a0322b-c09b-43ef-974a-771a7a66fc47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:28:40.575773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.575740    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh" pod "87a0322b-c09b-43ef-974a-771a7a66fc47" (UID: "87a0322b-c09b-43ef-974a-771a7a66fc47"). InnerVolumeSpecName "kube-api-access-jzjmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:40.672594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.672507    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:40.672594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.672542    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a0322b-c09b-43ef-974a-771a7a66fc47-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:40.672594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.672551    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a0322b-c09b-43ef-974a-771a7a66fc47-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:40.672594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:40.672561    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjmh\" (UniqueName: \"kubernetes.io/projected/87a0322b-c09b-43ef-974a-771a7a66fc47-kube-api-access-jzjmh\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:41.030251 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.030214    4928 generic.go:356] "Generic (PLEG): container finished" podID="87a0322b-c09b-43ef-974a-771a7a66fc47" containerID="b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3" exitCode=0
Nov 02 13:28:41.030704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.030261    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"
Nov 02 13:28:41.030704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.030295    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" event={"ID":"87a0322b-c09b-43ef-974a-771a7a66fc47","Type":"ContainerDied","Data":"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"}
Nov 02 13:28:41.030704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.030350    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg" event={"ID":"87a0322b-c09b-43ef-974a-771a7a66fc47","Type":"ContainerDied","Data":"bacdc48cf965c378b34ad4a923228fa7d2b00200a9912570cee0792b792d77d7"}
Nov 02 13:28:41.030704 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.030376    4928 scope.go:122] "RemoveContainer" containerID="b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"
Nov 02 13:28:41.042790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.042768    4928 scope.go:122] "RemoveContainer" containerID="b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"
Nov 02 13:28:41.043098 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:28:41.043075    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3\": container with ID starting with b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3 not found: ID does not exist" containerID="b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"
Nov 02 13:28:41.043206 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.043103    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3"} err="failed to get container status \"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3\": rpc error: code = NotFound desc = could not find container \"b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3\": container with ID starting with b4977582d159abbd7ea339b4bbdc1b5151bd63051fccf8086726dc9d313e44c3 not found: ID does not exist"
Nov 02 13:28:41.089196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.089158    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:28:41.101297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:41.101262    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef5efe-665d5f8488-zcqmg"]
Nov 02 13:28:42.499019 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:42.498983    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a0322b-c09b-43ef-974a-771a7a66fc47" path="/var/lib/kubelet/pods/87a0322b-c09b-43ef-974a-771a7a66fc47/volumes"
Nov 02 13:28:51.080736 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:51.080697    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:28:51.081201 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:51.080980    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy" containerID="cri-o://d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8" gracePeriod=30
Nov 02 13:28:56.365754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.365678    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:28:56.431707 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.431648    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.431751    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.431796    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.431887    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.431917    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432099 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.432079    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.432148    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.432187    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.432372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.432245    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\") pod \"7381c74f-d296-49a1-9f74-66f1838f7c1b\" (UID: \"7381c74f-d296-49a1-9f74-66f1838f7c1b\") "
Nov 02 13:28:56.433757 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.433710    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:56.434076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.434052    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:56.435078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.435050    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:56.435751 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.435720    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:56.435996 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.435969    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:56.436314 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.436288    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:56.438644 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.438604    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:28:56.438949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.438912    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "kube-api-access-9d4lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:56.440264 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.440238    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token" pod "7381c74f-d296-49a1-9f74-66f1838f7c1b" (UID: "7381c74f-d296-49a1-9f74-66f1838f7c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:56.534513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534470    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534513    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7381c74f-d296-49a1-9f74-66f1838f7c1b-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534532    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534545    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534561    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534575    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534589    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4lx\" (UniqueName: \"kubernetes.io/projected/7381c74f-d296-49a1-9f74-66f1838f7c1b-kube-api-access-9d4lx\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534603    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7381c74f-d296-49a1-9f74-66f1838f7c1b-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:56.534712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:56.534619    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7381c74f-d296-49a1-9f74-66f1838f7c1b-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:28:57.186630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.186596    4928 generic.go:356] "Generic (PLEG): container finished" podID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerID="d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8" exitCode=0
Nov 02 13:28:57.186851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.186649    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"
Nov 02 13:28:57.186851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.186678    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerDied","Data":"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"}
Nov 02 13:28:57.186851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.186743    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh" event={"ID":"7381c74f-d296-49a1-9f74-66f1838f7c1b","Type":"ContainerDied","Data":"008b68326efa6e0003bac559c86d18c5e4df816d81aae380ee0cc9fb5b2c4c4d"}
Nov 02 13:28:57.186851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.186771    4928 scope.go:122] "RemoveContainer" containerID="d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"
Nov 02 13:28:57.214840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.214607    4928 scope.go:122] "RemoveContainer" containerID="afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.231072    4928 scope.go:122] "RemoveContainer" containerID="d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:28:57.231511    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8\": container with ID starting with d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8 not found: ID does not exist" containerID="d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.231548    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8"} err="failed to get container status \"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8\": rpc error: code = NotFound desc = could not find container \"d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8\": container with ID starting with d5324d92d806dc24e34bceb0dd1b60e79cf4800534c5252baacc5b7840b4e3f8 not found: ID does not exist"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.231569    4928 scope.go:122] "RemoveContainer" containerID="afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:28:57.231919    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f\": container with ID starting with afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f not found: ID does not exist" containerID="afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"
Nov 02 13:28:57.238031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.232026    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f"} err="failed to get container status \"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f\": rpc error: code = NotFound desc = could not find container \"afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f\": container with ID starting with afe0fa4ee235c6519eaf10ff291e93c3c52f17cf10c716bc48abff7383ca9e8f not found: ID does not exist"
Nov 02 13:28:57.260798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.260764    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:28:57.273291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:57.273251    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-rlbdh"]
Nov 02 13:28:58.498549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:28:58.498513    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" path="/var/lib/kubelet/pods/7381c74f-d296-49a1-9f74-66f1838f7c1b/volumes"
Nov 02 13:29:09.730414 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:29:09.730363    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:29:39.730322 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:29:39.730276    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:29:51.821444 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:29:51.821398    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 13:29:59.894432 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:29:59.894394    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 13:29:59.941083 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:29:59.941045    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 13:30:00.220158 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.220066    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:00.221388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221363    4928 cpu_manager.go:414] "containerMap: removing container" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" containerName="maas-api"
Nov 02 13:30:00.221561 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221547    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" containerName="maas-api"
Nov 02 13:30:00.221691 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221681    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.221769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221759    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.221856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221845    4928 cpu_manager.go:414] "containerMap: removing container" podUID="797f5096-ae6a-4454-a351-25751edae1af" containerName="maas-api"
Nov 02 13:30:00.221962 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.221950    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797f5096-ae6a-4454-a351-25751edae1af" containerName="maas-api"
Nov 02 13:30:00.222062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222051    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.222135 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222125    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.222226 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222218    4928 cpu_manager.go:414] "containerMap: removing container" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.222298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222289    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.222381 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222373    4928 cpu_manager.go:414] "containerMap: removing container" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.222456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222447    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.222532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222523    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.222601 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222592    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.222682 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222672    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8b646e44-487a-4ea8-a520-6c7a12f772b4" containerName="maas-api"
Nov 02 13:30:00.222752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222744    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b646e44-487a-4ea8-a520-6c7a12f772b4" containerName="maas-api"
Nov 02 13:30:00.222842 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222820    4928 cpu_manager.go:414] "containerMap: removing container" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.222941 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222917    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222961    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222969    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222985    4928 cpu_manager.go:414] "containerMap: removing container" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.222993    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223011    4928 cpu_manager.go:414] "containerMap: removing container" podUID="87a0322b-c09b-43ef-974a-771a7a66fc47" containerName="maas-api"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223018    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a0322b-c09b-43ef-974a-771a7a66fc47" containerName="maas-api"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223031    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223038    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223059    4928 cpu_manager.go:414] "containerMap: removing container" podUID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" containerName="payload-processing"
Nov 02 13:30:00.223087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223066    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" containerName="payload-processing"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223258    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa0c403-890b-4006-ad32-eac4cb160996" containerName="maas-api"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223275    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223285    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="797f5096-ae6a-4454-a351-25751edae1af" containerName="maas-api"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223298    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223308    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223319    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="af290eea-132b-4d9a-83f0-1edbc5aaca2a" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223327    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223336    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223345    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b646e44-487a-4ea8-a520-6c7a12f772b4" containerName="maas-api"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223353    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223366    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53378b8-f9ab-4c5a-83dc-f6bed24b7aab" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223376    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223385    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="107a0384-09e7-42d3-902e-3f506d80c2fb" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223393    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a0322b-c09b-43ef-974a-771a7a66fc47" containerName="maas-api"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223403    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.223648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.223411    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae833f6-01fa-4f9f-b85e-44730039ae2e" containerName="payload-processing"
Nov 02 13:30:00.227881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.227862    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.254837 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.254788    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:30:00.267887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.267860    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:00.270104 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270081    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.270267 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270253    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344f928-7931-47b9-b15a-58fc25d6e878" containerName="istio-proxy"
Nov 02 13:30:00.270408 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270395    4928 cpu_manager.go:414] "containerMap: removing container" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.270504 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270493    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bef96c2-d223-4cd4-bc4f-8df0a100184b" containerName="istio-proxy"
Nov 02 13:30:00.270641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270628    4928 cpu_manager.go:414] "containerMap: removing container" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.270742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.270730    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.271060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.271045    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="7381c74f-d296-49a1-9f74-66f1838f7c1b" containerName="istio-proxy"
Nov 02 13:30:00.274898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.274874    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:00.274988 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.274978    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.282013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.281793    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.284128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.284103    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:00.291618 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291587    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hkwq\" (UniqueName: \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.291703 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291658    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d44b\" (UniqueName: \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.291754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291700    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.291893 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291873    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.291945 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291913    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk84x\" (UniqueName: \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.291981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.291961    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.311937 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.311908    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:00.318307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.318243    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:00.320374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.320346    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:00.326602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.326581    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.344376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.344293    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:00.359148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.359121    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:00.360210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.360172    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.381706 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.381680    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:00.393666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.393633    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66bl\" (UniqueName: \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.393847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.393689    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.393847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.393732    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.393847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.393791    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk84x\" (UniqueName: \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.394053 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.393881    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.394053 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394036    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkwq\" (UniqueName: \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.394173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394093    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d44b\" (UniqueName: \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.394173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394134    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.394346 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394167    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.394346 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394250    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.394474 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.394404    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.395384 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.395357    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.395548 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.395522    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.421528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.421502    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d44b\" (UniqueName: \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.421669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.421571    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkwq\" (UniqueName: \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.421728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.421702    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk84x\" (UniqueName: \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.496061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.496026    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66bl\" (UniqueName: \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.496261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.496089    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.496261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.496242    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.496387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.496285    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.496887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.496860    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.497096 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.497072    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.522731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.522696    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.522880 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.522795    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66bl\" (UniqueName: \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:00.547487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.547457    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:00.599330 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.599297    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:00.608726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.608618    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:00.661974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.654059    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:00.717853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:00.690053    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:01.066841 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.065081    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:01.067296 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:30:01.066962    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10322f0b_aba5_425a_a3a8_624622fa3689.slice/crio-e48a264312ade58610d5c3e9a35bfda3c8eaea65ebb6d188b792447c0edaeb0d WatchSource:0}: Error finding container e48a264312ade58610d5c3e9a35bfda3c8eaea65ebb6d188b792447c0edaeb0d: Status 404 returned error can't find the container with id e48a264312ade58610d5c3e9a35bfda3c8eaea65ebb6d188b792447c0edaeb0d
Nov 02 13:30:01.148697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.148661    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:01.157025 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:30:01.156990    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe9fac3_d823_4655_a6eb_27f6235d35e8.slice/crio-364e90a59ba5efaf0408797ee31f40c1177e74eb18d55776547463a9a4b8439b WatchSource:0}: Error finding container 364e90a59ba5efaf0408797ee31f40c1177e74eb18d55776547463a9a4b8439b: Status 404 returned error can't find the container with id 364e90a59ba5efaf0408797ee31f40c1177e74eb18d55776547463a9a4b8439b
Nov 02 13:30:01.193434 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.193404    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:01.203746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.203717    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:01.208469 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.208429    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:01.785337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.785304    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerStarted","Data":"3d9881a51402b65cd47de9cc386ae2c477f733673815666c81f3ffae4461afa2"}
Nov 02 13:30:01.785563 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.785364    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerStarted","Data":"364e90a59ba5efaf0408797ee31f40c1177e74eb18d55776547463a9a4b8439b"}
Nov 02 13:30:01.786730 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.786695    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerStarted","Data":"81e02d0051920027dbedfcd2ce925f6447158912629c3ca5909915e7411d6632"}
Nov 02 13:30:01.788223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.788193    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerStarted","Data":"7a5481d19d7ea65585479c8fb104b962fc90d0f32035ce96a667c07635de0854"}
Nov 02 13:30:01.788324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.788250    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerStarted","Data":"e48a264312ade58610d5c3e9a35bfda3c8eaea65ebb6d188b792447c0edaeb0d"}
Nov 02 13:30:01.789525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.789501    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerStarted","Data":"a656ddcc851923d7892dac5c29c5c550abd722148a1bd1449de9025ddebdce77"}
Nov 02 13:30:01.790973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.790945    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerStarted","Data":"8a6b2c7e269b26c93fad32262ed63d6113f2f0d9af6027a46483dfd513983001"}
Nov 02 13:30:01.791085 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.790986    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerStarted","Data":"09ebb76205e6928f0ee6105141e39c2ce5abf406745f1c2fdc1181a5bc3ca89d"}
Nov 02 13:30:01.833734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.833691    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" podStartSLOduration=1.833678148 podStartE2EDuration="1.833678148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.827840492 +0000 UTC m=+3799.476894442" watchObservedRunningTime="2026-11-02 13:30:01.833678148 +0000 UTC m=+3799.482732102"
Nov 02 13:30:01.872680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.872615    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" podStartSLOduration=1.87259592 podStartE2EDuration="1.87259592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.869907303 +0000 UTC m=+3799.518961259" watchObservedRunningTime="2026-11-02 13:30:01.87259592 +0000 UTC m=+3799.521649879"
Nov 02 13:30:01.915401 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:01.915292    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" podStartSLOduration=1.915279218 podStartE2EDuration="1.915279218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.912019995 +0000 UTC m=+3799.561073943" watchObservedRunningTime="2026-11-02 13:30:01.915279218 +0000 UTC m=+3799.564333218"
Nov 02 13:30:02.801540 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.801499    4928 generic.go:356] "Generic (PLEG): container finished" podID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerID="89a2071814d18771549a5a2266b6265a818bb6d1385d3ec7eb4f34a6639bfb9f" exitCode=60
Nov 02 13:30:02.802038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.801544    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerDied","Data":"89a2071814d18771549a5a2266b6265a818bb6d1385d3ec7eb4f34a6639bfb9f"}
Nov 02 13:30:02.802038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.801941    4928 scope.go:122] "RemoveContainer" containerID="89a2071814d18771549a5a2266b6265a818bb6d1385d3ec7eb4f34a6639bfb9f"
Nov 02 13:30:02.803524 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.803501    4928 generic.go:356] "Generic (PLEG): container finished" podID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerID="8a6b2c7e269b26c93fad32262ed63d6113f2f0d9af6027a46483dfd513983001" exitCode=60
Nov 02 13:30:02.803622 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.803585    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerDied","Data":"8a6b2c7e269b26c93fad32262ed63d6113f2f0d9af6027a46483dfd513983001"}
Nov 02 13:30:02.803986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.803881    4928 scope.go:122] "RemoveContainer" containerID="8a6b2c7e269b26c93fad32262ed63d6113f2f0d9af6027a46483dfd513983001"
Nov 02 13:30:02.805446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.805422    4928 generic.go:356] "Generic (PLEG): container finished" podID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerID="3d9881a51402b65cd47de9cc386ae2c477f733673815666c81f3ffae4461afa2" exitCode=60
Nov 02 13:30:02.805560 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.805504    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerDied","Data":"3d9881a51402b65cd47de9cc386ae2c477f733673815666c81f3ffae4461afa2"}
Nov 02 13:30:02.805752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.805730    4928 scope.go:122] "RemoveContainer" containerID="3d9881a51402b65cd47de9cc386ae2c477f733673815666c81f3ffae4461afa2"
Nov 02 13:30:02.807237 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.807213    4928 generic.go:356] "Generic (PLEG): container finished" podID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerID="cf8d62671fc696ed243f67ed80398c281d242f5ea70c037bfddc01bf4f73177e" exitCode=60
Nov 02 13:30:02.807335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.807292    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerDied","Data":"cf8d62671fc696ed243f67ed80398c281d242f5ea70c037bfddc01bf4f73177e"}
Nov 02 13:30:02.807592 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.807537    4928 scope.go:122] "RemoveContainer" containerID="cf8d62671fc696ed243f67ed80398c281d242f5ea70c037bfddc01bf4f73177e"
Nov 02 13:30:02.809036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.809012    4928 generic.go:356] "Generic (PLEG): container finished" podID="10322f0b-aba5-425a-a3a8-624622fa3689" containerID="7a5481d19d7ea65585479c8fb104b962fc90d0f32035ce96a667c07635de0854" exitCode=60
Nov 02 13:30:02.809130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.809053    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerDied","Data":"7a5481d19d7ea65585479c8fb104b962fc90d0f32035ce96a667c07635de0854"}
Nov 02 13:30:02.809278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:02.809264    4928 scope.go:122] "RemoveContainer" containerID="7a5481d19d7ea65585479c8fb104b962fc90d0f32035ce96a667c07635de0854"
Nov 02 13:30:03.835161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.835119    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerStarted","Data":"a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"}
Nov 02 13:30:03.839979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.839630    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerStarted","Data":"b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"}
Nov 02 13:30:03.842291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.842257    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerStarted","Data":"56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"}
Nov 02 13:30:03.844335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.844306    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerStarted","Data":"a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"}
Nov 02 13:30:03.846512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.846485    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerStarted","Data":"0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"}
Nov 02 13:30:03.888661 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.888608    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" podStartSLOduration=3.888595833 podStartE2EDuration="3.888595833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.887722931 +0000 UTC m=+3801.536776892" watchObservedRunningTime="2026-11-02 13:30:03.888595833 +0000 UTC m=+3801.537649792"
Nov 02 13:30:03.949089 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:03.949028    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" podStartSLOduration=3.949009669 podStartE2EDuration="3.949009669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.941419813 +0000 UTC m=+3801.590473773" watchObservedRunningTime="2026-11-02 13:30:03.949009669 +0000 UTC m=+3801.598063629"
Nov 02 13:30:04.856514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.856475    4928 generic.go:356] "Generic (PLEG): container finished" podID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerID="56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449" exitCode=60
Nov 02 13:30:04.857068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.856556    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerDied","Data":"56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"}
Nov 02 13:30:04.857068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.856632    4928 scope.go:122] "RemoveContainer" containerID="3d9881a51402b65cd47de9cc386ae2c477f733673815666c81f3ffae4461afa2"
Nov 02 13:30:04.857068 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.856953    4928 scope.go:122] "RemoveContainer" containerID="56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"
Nov 02 13:30:04.857372 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:04.857343    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6_odh-ai-gateway-infra(1fe9fac3-d823-4655-a6eb-27f6235d35e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8"
Nov 02 13:30:04.859869 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.859812    4928 generic.go:356] "Generic (PLEG): container finished" podID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerID="a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128" exitCode=60
Nov 02 13:30:04.860006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.859883    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerDied","Data":"a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"}
Nov 02 13:30:04.860190 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.860176    4928 scope.go:122] "RemoveContainer" containerID="a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"
Nov 02 13:30:04.860508 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:04.860485    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5_odh-ai-gateway-infra(9ca0717d-8b99-4ee1-86bd-25fa117c9d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d"
Nov 02 13:30:04.861959 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.861939    4928 generic.go:356] "Generic (PLEG): container finished" podID="10322f0b-aba5-425a-a3a8-624622fa3689" containerID="0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd" exitCode=60
Nov 02 13:30:04.862073 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.862022    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerDied","Data":"0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"}
Nov 02 13:30:04.862264 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.862250    4928 scope.go:122] "RemoveContainer" containerID="0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"
Nov 02 13:30:04.862579 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:04.862556    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt_odh-ai-gateway-infra(10322f0b-aba5-425a-a3a8-624622fa3689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" podUID="10322f0b-aba5-425a-a3a8-624622fa3689"
Nov 02 13:30:04.864218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.864199    4928 generic.go:356] "Generic (PLEG): container finished" podID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerID="a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e" exitCode=60
Nov 02 13:30:04.864312 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.864285    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerDied","Data":"a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"}
Nov 02 13:30:04.864528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.864509    4928 scope.go:122] "RemoveContainer" containerID="a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"
Nov 02 13:30:04.864810 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:04.864791    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq_odh-ai-gateway-infra(c81ba970-d3cd-49f1-9bad-d5e345f430d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5"
Nov 02 13:30:04.866626 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.866606    4928 generic.go:356] "Generic (PLEG): container finished" podID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerID="b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53" exitCode=60
Nov 02 13:30:04.866738 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.866635    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerDied","Data":"b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"}
Nov 02 13:30:04.866960 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.866943    4928 scope.go:122] "RemoveContainer" containerID="b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"
Nov 02 13:30:04.867209 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:04.867187    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5_odh-ai-gateway-infra(90bca6ff-5b72-4cc8-be02-badc173e9845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845"
Nov 02 13:30:04.871466 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.871446    4928 scope.go:122] "RemoveContainer" containerID="cf8d62671fc696ed243f67ed80398c281d242f5ea70c037bfddc01bf4f73177e"
Nov 02 13:30:04.885385 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.885362    4928 scope.go:122] "RemoveContainer" containerID="7a5481d19d7ea65585479c8fb104b962fc90d0f32035ce96a667c07635de0854"
Nov 02 13:30:04.894928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.894898    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing" containerID="cri-o://2044d0409ce35fb2ed2eb81112a1830203dc948b77652c6f91c4febb7b7f681c" gracePeriod=25
Nov 02 13:30:04.902799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.902761    4928 scope.go:122] "RemoveContainer" containerID="89a2071814d18771549a5a2266b6265a818bb6d1385d3ec7eb4f34a6639bfb9f"
Nov 02 13:30:04.918919 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.918885    4928 scope.go:122] "RemoveContainer" containerID="8a6b2c7e269b26c93fad32262ed63d6113f2f0d9af6027a46483dfd513983001"
Nov 02 13:30:04.941531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:04.941493    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing" containerID="cri-o://4882cca20dc362d6b02bc1ef2b7ca4ace978c233fafc9fc40cc79139c79bb27a" gracePeriod=25
Nov 02 13:30:05.415398 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.415356    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:30:05.415631 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.415601    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" podUID="067983aa-fbee-4750-83ae-3bbbda43fc0a" containerName="maas-api" containerID="cri-o://ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa" gracePeriod=30
Nov 02 13:30:05.808338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.808301    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:30:05.926538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.926506    4928 scope.go:122] "RemoveContainer" containerID="b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"
Nov 02 13:30:05.927818 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:05.927781    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5_odh-ai-gateway-infra(90bca6ff-5b72-4cc8-be02-badc173e9845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845"
Nov 02 13:30:05.934459 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.934434    4928 scope.go:122] "RemoveContainer" containerID="56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"
Nov 02 13:30:05.940276 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.940255    4928 scope.go:122] "RemoveContainer" containerID="a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"
Nov 02 13:30:05.940890 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:05.940862    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5_odh-ai-gateway-infra(9ca0717d-8b99-4ee1-86bd-25fa117c9d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d"
Nov 02 13:30:05.946007 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:05.945954    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6_odh-ai-gateway-infra(1fe9fac3-d823-4655-a6eb-27f6235d35e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8"
Nov 02 13:30:05.947273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.947246    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\") pod \"067983aa-fbee-4750-83ae-3bbbda43fc0a\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") "
Nov 02 13:30:05.947425 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.947409    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\") pod \"067983aa-fbee-4750-83ae-3bbbda43fc0a\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") "
Nov 02 13:30:05.947587 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.947573    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\" (UniqueName: \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\") pod \"067983aa-fbee-4750-83ae-3bbbda43fc0a\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") "
Nov 02 13:30:05.947844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.947806    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\") pod \"067983aa-fbee-4750-83ae-3bbbda43fc0a\" (UID: \"067983aa-fbee-4750-83ae-3bbbda43fc0a\") "
Nov 02 13:30:05.948791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.948770    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca" pod "067983aa-fbee-4750-83ae-3bbbda43fc0a" (UID: "067983aa-fbee-4750-83ae-3bbbda43fc0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:05.953579 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.953559    4928 scope.go:122] "RemoveContainer" containerID="0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"
Nov 02 13:30:05.954024 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:05.953997    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt_odh-ai-gateway-infra(10322f0b-aba5-425a-a3a8-624622fa3689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" podUID="10322f0b-aba5-425a-a3a8-624622fa3689"
Nov 02 13:30:05.955404 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.955378    4928 scope.go:122] "RemoveContainer" containerID="a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"
Nov 02 13:30:05.955668 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:05.955640    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq_odh-ai-gateway-infra(c81ba970-d3cd-49f1-9bad-d5e345f430d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5"
Nov 02 13:30:05.956302 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.956274    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4" pod "067983aa-fbee-4750-83ae-3bbbda43fc0a" (UID: "067983aa-fbee-4750-83ae-3bbbda43fc0a"). InnerVolumeSpecName "kube-api-access-pm2w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:05.957421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.957396    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls" pod "067983aa-fbee-4750-83ae-3bbbda43fc0a" (UID: "067983aa-fbee-4750-83ae-3bbbda43fc0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:05.958086 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958063    4928 generic.go:356] "Generic (PLEG): container finished" podID="067983aa-fbee-4750-83ae-3bbbda43fc0a" containerID="ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa" exitCode=0
Nov 02 13:30:05.958189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958128    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" event={"ID":"067983aa-fbee-4750-83ae-3bbbda43fc0a","Type":"ContainerDied","Data":"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"}
Nov 02 13:30:05.958257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958206    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x" event={"ID":"067983aa-fbee-4750-83ae-3bbbda43fc0a","Type":"ContainerDied","Data":"a80fb15939f333ae457e60a3801ca2afde14d7158f4b79a61e82f2ad34ff542d"}
Nov 02 13:30:05.958257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958238    4928 scope.go:122] "RemoveContainer" containerID="ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"
Nov 02 13:30:05.958403 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958373    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"
Nov 02 13:30:05.958973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:05.958953    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs" pod "067983aa-fbee-4750-83ae-3bbbda43fc0a" (UID: "067983aa-fbee-4750-83ae-3bbbda43fc0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:06.001640 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.001615    4928 scope.go:122] "RemoveContainer" containerID="ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"
Nov 02 13:30:06.002314 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:06.002274    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa\": container with ID starting with ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa not found: ID does not exist" containerID="ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"
Nov 02 13:30:06.002460 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.002308    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa"} err="failed to get container status \"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa\": rpc error: code = NotFound desc = could not find container \"ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa\": container with ID starting with ec9f9bf743084545432f2ca852ce2ca544ee17848dc7e610fb91dd52cd71e0fa not found: ID does not exist"
Nov 02 13:30:06.051153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.051075    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:06.051153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.051102    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067983aa-fbee-4750-83ae-3bbbda43fc0a-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:06.051153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.051128    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2w4\" (UniqueName: \"kubernetes.io/projected/067983aa-fbee-4750-83ae-3bbbda43fc0a-kube-api-access-pm2w4\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:06.051153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.051138    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067983aa-fbee-4750-83ae-3bbbda43fc0a-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:06.320272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.320183    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:30:06.349181 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.349142    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-00d004-7c7664795f-qzt8x"]
Nov 02 13:30:06.498749 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.498710    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067983aa-fbee-4750-83ae-3bbbda43fc0a" path="/var/lib/kubelet/pods/067983aa-fbee-4750-83ae-3bbbda43fc0a/volumes"
Nov 02 13:30:06.843740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.843697    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main" containerID="cri-o://33957e8a6312ccc89a4b8f8e13f51d67905da99dea7c7f6c5fc720bd494ba128" gracePeriod=45
Nov 02 13:30:06.968438 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.968406    4928 generic.go:356] "Generic (PLEG): container finished" podID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerID="33957e8a6312ccc89a4b8f8e13f51d67905da99dea7c7f6c5fc720bd494ba128" exitCode=0
Nov 02 13:30:06.968841 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:06.968478    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" event={"ID":"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06","Type":"ContainerDied","Data":"33957e8a6312ccc89a4b8f8e13f51d67905da99dea7c7f6c5fc720bd494ba128"}
Nov 02 13:30:07.135573 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.135542    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 13:30:07.268728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.268697    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.268975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.268737    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.268975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.268805    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.268975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.268890    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\" (UniqueName: \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.268975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.268926    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.269212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269064    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.269212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269093    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\") pod \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\" (UID: \"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06\") "
Nov 02 13:30:07.269212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269118    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.269353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269341    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.269738 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269717    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.269870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269740    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.269870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.269755    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.272023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.271994    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:07.272159 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.271909    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.272159 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.272041    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct" pod "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" (UID: "3a1e00ed-5726-4d6e-807e-8ac7cec0fc06"). InnerVolumeSpecName "kube-api-access-mdxct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:07.371155 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.371050    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.371155 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.371082    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxct\" (UniqueName: \"kubernetes.io/projected/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-kube-api-access-mdxct\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.371155 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.371094    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.980487 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.980454    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"
Nov 02 13:30:07.980944 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.980487    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn" event={"ID":"3a1e00ed-5726-4d6e-807e-8ac7cec0fc06","Type":"ContainerDied","Data":"7b234355ce650634aa61b821643c2e7e8655df953dd47c3cee4578682f685dd1"}
Nov 02 13:30:07.980944 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:07.980538    4928 scope.go:122] "RemoveContainer" containerID="33957e8a6312ccc89a4b8f8e13f51d67905da99dea7c7f6c5fc720bd494ba128"
Nov 02 13:30:08.038612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:08.038576    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 13:30:08.053740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:08.053712    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-2-w4-843978-kserve-fc6455c9c-r86vn"]
Nov 02 13:30:08.306777 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:08.306746    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:30:08.306976 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:08.306954    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" containerID="cri-o://13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc" gracePeriod=30
Nov 02 13:30:08.506890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:08.506844    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" path="/var/lib/kubelet/pods/3a1e00ed-5726-4d6e-807e-8ac7cec0fc06/volumes"
Nov 02 13:30:09.142786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:09.142743    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:09.729903 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:09.729863    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:30:10.151406 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:10.151363    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:12.384280 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:12.384239    4928 patch_prober.go:29] interesting pod/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:30:12.384724 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:12.384307    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:30:13.624600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.624571    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:30:13.785124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785082    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785144    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785183    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785288    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785325    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785360    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785430    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785447    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:13.785569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785511    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785540    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\") pod \"474cc566-a72a-4bf1-b668-eb05ae7985a8\" (UID: \"474cc566-a72a-4bf1-b668-eb05ae7985a8\") "
Nov 02 13:30:13.785938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785627    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:13.785938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785641    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:13.786063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.785958    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:13.786122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786080    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:13.786273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786248    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.786363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786280    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.786363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786298    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.786363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786315    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474cc566-a72a-4bf1-b668-eb05ae7985a8-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.786363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.786331    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.787720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.787691    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:13.787871 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.787724    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "kube-api-access-t9s8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:13.788091 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.788074    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:30:13.788143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.788096    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy" pod "474cc566-a72a-4bf1-b668-eb05ae7985a8" (UID: "474cc566-a72a-4bf1-b668-eb05ae7985a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:13.887623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.887532    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.887623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.887567    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.887623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.887576    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474cc566-a72a-4bf1-b668-eb05ae7985a8-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:13.887623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:13.887586    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9s8d\" (UniqueName: \"kubernetes.io/projected/474cc566-a72a-4bf1-b668-eb05ae7985a8-kube-api-access-t9s8d\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:14.058120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.058080    4928 generic.go:356] "Generic (PLEG): container finished" podID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerID="13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc" exitCode=0
Nov 02 13:30:14.058327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.058136    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"
Nov 02 13:30:14.058327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.058163    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerDied","Data":"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"}
Nov 02 13:30:14.058327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.058243    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt" event={"ID":"474cc566-a72a-4bf1-b668-eb05ae7985a8","Type":"ContainerDied","Data":"7797a5dc31f5d57dc99208380dbc01e05689a8fbd4c1be7c614d291898c499eb"}
Nov 02 13:30:14.058327 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.058272    4928 scope.go:122] "RemoveContainer" containerID="13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"
Nov 02 13:30:14.070996 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.070970    4928 scope.go:122] "RemoveContainer" containerID="3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"
Nov 02 13:30:14.081355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.081334    4928 scope.go:122] "RemoveContainer" containerID="13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"
Nov 02 13:30:14.081682 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:14.081659    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc\": container with ID starting with 13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc not found: ID does not exist" containerID="13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"
Nov 02 13:30:14.081730 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.081691    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc"} err="failed to get container status \"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc\": rpc error: code = NotFound desc = could not find container \"13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc\": container with ID starting with 13ce12aaa2776e979163c4354ef80b6ae219d5cf650be8bcf2b133132648f5cc not found: ID does not exist"
Nov 02 13:30:14.081730 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.081709    4928 scope.go:122] "RemoveContainer" containerID="3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"
Nov 02 13:30:14.081982 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:14.081961    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095\": container with ID starting with 3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095 not found: ID does not exist" containerID="3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"
Nov 02 13:30:14.082036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.081990    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095"} err="failed to get container status \"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095\": rpc error: code = NotFound desc = could not find container \"3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095\": container with ID starting with 3d51e2bc7edb70ca8291c5420ef8d709f1152882c88f25c44ad42e4b4673a095 not found: ID does not exist"
Nov 02 13:30:14.116454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.116426    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:30:14.134053 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.134021    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-00d004-openshift-default-56d4b77877-55tqt"]
Nov 02 13:30:14.490728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.490691    4928 scope.go:122] "RemoveContainer" containerID="b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"
Nov 02 13:30:14.501996 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:14.501965    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" path="/var/lib/kubelet/pods/474cc566-a72a-4bf1-b668-eb05ae7985a8/volumes"
Nov 02 13:30:15.080865 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:15.080806    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerStarted","Data":"f91b547ec800751d88e019bfe02c3e3074c198e05d825041ba0c97e45747163b"}
Nov 02 13:30:15.490243 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:15.490206    4928 scope.go:122] "RemoveContainer" containerID="0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"
Nov 02 13:30:16.091356 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.091253    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerStarted","Data":"039318e14542703629b4b6a65ac42817f2cf6af1bbe5e195b8f15ff8802c6336"}
Nov 02 13:30:16.093654 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.093626    4928 generic.go:356] "Generic (PLEG): container finished" podID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerID="f91b547ec800751d88e019bfe02c3e3074c198e05d825041ba0c97e45747163b" exitCode=60
Nov 02 13:30:16.093804 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.093655    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerDied","Data":"f91b547ec800751d88e019bfe02c3e3074c198e05d825041ba0c97e45747163b"}
Nov 02 13:30:16.093804 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.093718    4928 scope.go:122] "RemoveContainer" containerID="b8ba5742bcbad7f98e7e2be7a96f5cfe5d7b338cc2903b0975e40aea380d3b53"
Nov 02 13:30:16.094150 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.094134    4928 scope.go:122] "RemoveContainer" containerID="f91b547ec800751d88e019bfe02c3e3074c198e05d825041ba0c97e45747163b"
Nov 02 13:30:16.094505 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:16.094477    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5_odh-ai-gateway-infra(90bca6ff-5b72-4cc8-be02-badc173e9845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845"
Nov 02 13:30:16.158689 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.158655    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:16.490282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.490249    4928 scope.go:122] "RemoveContainer" containerID="56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"
Nov 02 13:30:16.737630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.737585    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 13:30:16.783364 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:16.783334    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 13:30:17.104622 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.104591    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerStarted","Data":"8eff1ec923c3c07a7cc0d101e57a55625d6b2f624824ea7c700ca38fa88462ad"}
Nov 02 13:30:17.106756 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.106731    4928 generic.go:356] "Generic (PLEG): container finished" podID="10322f0b-aba5-425a-a3a8-624622fa3689" containerID="039318e14542703629b4b6a65ac42817f2cf6af1bbe5e195b8f15ff8802c6336" exitCode=60
Nov 02 13:30:17.106878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.106797    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerDied","Data":"039318e14542703629b4b6a65ac42817f2cf6af1bbe5e195b8f15ff8802c6336"}
Nov 02 13:30:17.106939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.106902    4928 scope.go:122] "RemoveContainer" containerID="0f804f44df39d07093b09f9e2afd77c8fd20d4ca1260a7d4a6480ee9224356fd"
Nov 02 13:30:17.107301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.107282    4928 scope.go:122] "RemoveContainer" containerID="039318e14542703629b4b6a65ac42817f2cf6af1bbe5e195b8f15ff8802c6336"
Nov 02 13:30:17.107669 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:17.107646    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt_odh-ai-gateway-infra(10322f0b-aba5-425a-a3a8-624622fa3689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" podUID="10322f0b-aba5-425a-a3a8-624622fa3689"
Nov 02 13:30:17.183845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.169019    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:17.323365 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.323336    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:17.342575 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.342542    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\") pod \"90bca6ff-5b72-4cc8-be02-badc173e9845\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") "
Nov 02 13:30:17.342858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.342835    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\" (UniqueName: \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\") pod \"90bca6ff-5b72-4cc8-be02-badc173e9845\" (UID: \"90bca6ff-5b72-4cc8-be02-badc173e9845\") "
Nov 02 13:30:17.342991 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.342971    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca" pod "90bca6ff-5b72-4cc8-be02-badc173e9845" (UID: "90bca6ff-5b72-4cc8-be02-badc173e9845"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:17.343321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.343298    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bca6ff-5b72-4cc8-be02-badc173e9845-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:17.344867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.344839    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl" pod "90bca6ff-5b72-4cc8-be02-badc173e9845" (UID: "90bca6ff-5b72-4cc8-be02-badc173e9845"). InnerVolumeSpecName "kube-api-access-q66bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:17.464847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.457468    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66bl\" (UniqueName: \"kubernetes.io/projected/90bca6ff-5b72-4cc8-be02-badc173e9845-kube-api-access-q66bl\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:17.534692 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:17.532622    4928 scope.go:122] "RemoveContainer" containerID="a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"
Nov 02 13:30:18.115595 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.115558    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"
Nov 02 13:30:18.116107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.115625    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5" event={"ID":"90bca6ff-5b72-4cc8-be02-badc173e9845","Type":"ContainerDied","Data":"09ebb76205e6928f0ee6105141e39c2ce5abf406745f1c2fdc1181a5bc3ca89d"}
Nov 02 13:30:18.116107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.115686    4928 scope.go:122] "RemoveContainer" containerID="f91b547ec800751d88e019bfe02c3e3074c198e05d825041ba0c97e45747163b"
Nov 02 13:30:18.118260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.118042    4928 generic.go:356] "Generic (PLEG): container finished" podID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerID="8eff1ec923c3c07a7cc0d101e57a55625d6b2f624824ea7c700ca38fa88462ad" exitCode=60
Nov 02 13:30:18.118260 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.118122    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerDied","Data":"8eff1ec923c3c07a7cc0d101e57a55625d6b2f624824ea7c700ca38fa88462ad"}
Nov 02 13:30:18.118591 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.118561    4928 scope.go:122] "RemoveContainer" containerID="8eff1ec923c3c07a7cc0d101e57a55625d6b2f624824ea7c700ca38fa88462ad"
Nov 02 13:30:18.118997 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:18.118962    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6_odh-ai-gateway-infra(1fe9fac3-d823-4655-a6eb-27f6235d35e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8"
Nov 02 13:30:18.132060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.131880    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerStarted","Data":"0cf289f50ede9aab7d37b94e815bea6f7c2046c2707d85837f73ab195fcb0921"}
Nov 02 13:30:18.136533 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.136511    4928 scope.go:122] "RemoveContainer" containerID="56587a6c38a14a4aeb3f817fc518bf913ab0468df109979e611d9b66cb25f449"
Nov 02 13:30:18.234970 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.234256    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:18.312374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.312341    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:18.375377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.375283    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\" (UniqueName: \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\") pod \"10322f0b-aba5-425a-a3a8-624622fa3689\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") "
Nov 02 13:30:18.375567 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.375411    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\") pod \"10322f0b-aba5-425a-a3a8-624622fa3689\" (UID: \"10322f0b-aba5-425a-a3a8-624622fa3689\") "
Nov 02 13:30:18.375815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.375785    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca" pod "10322f0b-aba5-425a-a3a8-624622fa3689" (UID: "10322f0b-aba5-425a-a3a8-624622fa3689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:18.376064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.376045    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10322f0b-aba5-425a-a3a8-624622fa3689-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.377616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.377586    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b" pod "10322f0b-aba5-425a-a3a8-624622fa3689" (UID: "10322f0b-aba5-425a-a3a8-624622fa3689"). InnerVolumeSpecName "kube-api-access-6d44b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:18.473358 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.473324    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:18.477022 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.476987    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d44b\" (UniqueName: \"kubernetes.io/projected/10322f0b-aba5-425a-a3a8-624622fa3689-kube-api-access-6d44b\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.483449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.483422    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893770-qlgn5"]
Nov 02 13:30:18.490232 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.490204    4928 scope.go:122] "RemoveContainer" containerID="a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"
Nov 02 13:30:18.499244 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:18.499218    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" path="/var/lib/kubelet/pods/90bca6ff-5b72-4cc8-be02-badc173e9845/volumes"
Nov 02 13:30:19.141428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.141378    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:19.144612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.144577    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerStarted","Data":"30c742c67660ab03ecf5e9a51675b2e1d68e3ca80e5593ef4651d9308746f166"}
Nov 02 13:30:19.146333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.146312    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"
Nov 02 13:30:19.146448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.146355    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt" event={"ID":"10322f0b-aba5-425a-a3a8-624622fa3689","Type":"ContainerDied","Data":"e48a264312ade58610d5c3e9a35bfda3c8eaea65ebb6d188b792447c0edaeb0d"}
Nov 02 13:30:19.146448 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.146410    4928 scope.go:122] "RemoveContainer" containerID="039318e14542703629b4b6a65ac42817f2cf6af1bbe5e195b8f15ff8802c6336"
Nov 02 13:30:19.148726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.148700    4928 generic.go:356] "Generic (PLEG): container finished" podID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerID="0cf289f50ede9aab7d37b94e815bea6f7c2046c2707d85837f73ab195fcb0921" exitCode=60
Nov 02 13:30:19.148931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.148770    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerDied","Data":"0cf289f50ede9aab7d37b94e815bea6f7c2046c2707d85837f73ab195fcb0921"}
Nov 02 13:30:19.149205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.149186    4928 scope.go:122] "RemoveContainer" containerID="0cf289f50ede9aab7d37b94e815bea6f7c2046c2707d85837f73ab195fcb0921"
Nov 02 13:30:19.149527 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:19.149503    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq_odh-ai-gateway-infra(c81ba970-d3cd-49f1-9bad-d5e345f430d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5"
Nov 02 13:30:19.159236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.159208    4928 scope.go:122] "RemoveContainer" containerID="a55c801ac1816f262dd5ef1d57c6487fd4b85cae4d83c4658c1cd5cf5ae2fa2e"
Nov 02 13:30:19.319515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.319484    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:19.325219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.325181    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:19.333903 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.333874    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893770-ch8jt"]
Nov 02 13:30:19.390739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.390706    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\") pod \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") "
Nov 02 13:30:19.390911 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.390807    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\") pod \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\" (UID: \"1fe9fac3-d823-4655-a6eb-27f6235d35e8\") "
Nov 02 13:30:19.391200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.391171    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca" pod "1fe9fac3-d823-4655-a6eb-27f6235d35e8" (UID: "1fe9fac3-d823-4655-a6eb-27f6235d35e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:19.391466 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.391418    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe9fac3-d823-4655-a6eb-27f6235d35e8-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:19.393293 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.393269    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7" pod "1fe9fac3-d823-4655-a6eb-27f6235d35e8" (UID: "1fe9fac3-d823-4655-a6eb-27f6235d35e8"). InnerVolumeSpecName "kube-api-access-dtpx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:19.438630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.438592    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:19.498852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:19.494978    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpx7\" (UniqueName: \"kubernetes.io/projected/1fe9fac3-d823-4655-a6eb-27f6235d35e8-kube-api-access-dtpx7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.150496 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.150454    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:20.161031 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.161004    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"
Nov 02 13:30:20.161194 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.161024    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6" event={"ID":"1fe9fac3-d823-4655-a6eb-27f6235d35e8","Type":"ContainerDied","Data":"364e90a59ba5efaf0408797ee31f40c1177e74eb18d55776547463a9a4b8439b"}
Nov 02 13:30:20.161194 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.161109    4928 scope.go:122] "RemoveContainer" containerID="8eff1ec923c3c07a7cc0d101e57a55625d6b2f624824ea7c700ca38fa88462ad"
Nov 02 13:30:20.163599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.163573    4928 generic.go:356] "Generic (PLEG): container finished" podID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerID="30c742c67660ab03ecf5e9a51675b2e1d68e3ca80e5593ef4651d9308746f166" exitCode=60
Nov 02 13:30:20.163722 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.163651    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerDied","Data":"30c742c67660ab03ecf5e9a51675b2e1d68e3ca80e5593ef4651d9308746f166"}
Nov 02 13:30:20.164172 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.164151    4928 scope.go:122] "RemoveContainer" containerID="30c742c67660ab03ecf5e9a51675b2e1d68e3ca80e5593ef4651d9308746f166"
Nov 02 13:30:20.164497 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:20.164473    4928 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5_odh-ai-gateway-infra(9ca0717d-8b99-4ee1-86bd-25fa117c9d1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d"
Nov 02 13:30:20.175061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.175013    4928 scope.go:122] "RemoveContainer" containerID="a10cae81fe3bb56f005f9f0317fdc0be63327c0c6ff47c938fdad22154d25128"
Nov 02 13:30:20.248881 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.247194    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:20.297892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.297856    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:20.306477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.306439    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893770-cvhz6"]
Nov 02 13:30:20.332470 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.332443    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:20.406635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.406532    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\" (UniqueName: \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\") pod \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") "
Nov 02 13:30:20.406635 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.406605    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\") pod \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\" (UID: \"c81ba970-d3cd-49f1-9bad-d5e345f430d5\") "
Nov 02 13:30:20.407084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.407050    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca" pod "c81ba970-d3cd-49f1-9bad-d5e345f430d5" (UID: "c81ba970-d3cd-49f1-9bad-d5e345f430d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.407196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.407162    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81ba970-d3cd-49f1-9bad-d5e345f430d5-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.408803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.408778    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq" pod "c81ba970-d3cd-49f1-9bad-d5e345f430d5" (UID: "c81ba970-d3cd-49f1-9bad-d5e345f430d5"). InnerVolumeSpecName "kube-api-access-7hkwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.498291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.498254    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" path="/var/lib/kubelet/pods/10322f0b-aba5-425a-a3a8-624622fa3689/volumes"
Nov 02 13:30:20.498876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.498855    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" path="/var/lib/kubelet/pods/1fe9fac3-d823-4655-a6eb-27f6235d35e8/volumes"
Nov 02 13:30:20.508171 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:20.508147    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hkwq\" (UniqueName: \"kubernetes.io/projected/c81ba970-d3cd-49f1-9bad-d5e345f430d5-kube-api-access-7hkwq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.179484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.179363    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"
Nov 02 13:30:21.179484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.179366    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq" event={"ID":"c81ba970-d3cd-49f1-9bad-d5e345f430d5","Type":"ContainerDied","Data":"a656ddcc851923d7892dac5c29c5c550abd722148a1bd1449de9025ddebdce77"}
Nov 02 13:30:21.179484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.179470    4928 scope.go:122] "RemoveContainer" containerID="0cf289f50ede9aab7d37b94e815bea6f7c2046c2707d85837f73ab195fcb0921"
Nov 02 13:30:21.293779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.293742    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:21.297712 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.297683    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893770-8flqq"]
Nov 02 13:30:21.418757 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.418731    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:21.525013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.524977    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\" (UniqueName: \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\") pod \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") "
Nov 02 13:30:21.525230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.525039    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\") pod \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\" (UID: \"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d\") "
Nov 02 13:30:21.525502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.525471    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca" pod "9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" (UID: "9ca0717d-8b99-4ee1-86bd-25fa117c9d1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.527353 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.527327    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x" pod "9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" (UID: "9ca0717d-8b99-4ee1-86bd-25fa117c9d1d"). InnerVolumeSpecName "kube-api-access-sk84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.626794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.626756    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-maas-api-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.626794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.626792    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk84x\" (UniqueName: \"kubernetes.io/projected/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d-kube-api-access-sk84x\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.739240 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.739189    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing" containerID="cri-o://2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a" gracePeriod=25
Nov 02 13:30:21.784503 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:21.784412    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing" containerID="cri-o://295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f" gracePeriod=25
Nov 02 13:30:22.188591 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.188507    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"
Nov 02 13:30:22.189113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.188504    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5" event={"ID":"9ca0717d-8b99-4ee1-86bd-25fa117c9d1d","Type":"ContainerDied","Data":"81e02d0051920027dbedfcd2ce925f6447158912629c3ca5909915e7411d6632"}
Nov 02 13:30:22.189113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.188650    4928 scope.go:122] "RemoveContainer" containerID="30c742c67660ab03ecf5e9a51675b2e1d68e3ca80e5593ef4651d9308746f166"
Nov 02 13:30:22.244612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.244572    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:22.257764 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.257733    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893770-8q9c5"]
Nov 02 13:30:22.500042 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.500007    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" path="/var/lib/kubelet/pods/9ca0717d-8b99-4ee1-86bd-25fa117c9d1d/volumes"
Nov 02 13:30:22.500511 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:22.500488    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" path="/var/lib/kubelet/pods/c81ba970-d3cd-49f1-9bad-d5e345f430d5/volumes"
Nov 02 13:30:25.237232 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.237193    4928 generic.go:356] "Generic (PLEG): container finished" podID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerID="2044d0409ce35fb2ed2eb81112a1830203dc948b77652c6f91c4febb7b7f681c" exitCode=0
Nov 02 13:30:25.237697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.237364    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" event={"ID":"14a9f87a-1272-4591-bd5d-df59619b1f28","Type":"ContainerDied","Data":"2044d0409ce35fb2ed2eb81112a1830203dc948b77652c6f91c4febb7b7f681c"}
Nov 02 13:30:25.237697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.237434    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7" event={"ID":"14a9f87a-1272-4591-bd5d-df59619b1f28","Type":"ContainerDied","Data":"ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100"}
Nov 02 13:30:25.237697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.237462    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac24b839ab2799f8a533ae1c2287666f871f2e65f191a4eea566fa25eb0f7100"
Nov 02 13:30:25.240011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.239977    4928 generic.go:356] "Generic (PLEG): container finished" podID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerID="4882cca20dc362d6b02bc1ef2b7ca4ace978c233fafc9fc40cc79139c79bb27a" exitCode=0
Nov 02 13:30:25.240145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.240067    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" event={"ID":"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25","Type":"ContainerDied","Data":"4882cca20dc362d6b02bc1ef2b7ca4ace978c233fafc9fc40cc79139c79bb27a"}
Nov 02 13:30:25.241630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.241607    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 13:30:25.318325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.318298    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 13:30:25.377378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.377341    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\") pod \"14a9f87a-1272-4591-bd5d-df59619b1f28\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") "
Nov 02 13:30:25.377576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.377468    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\") pod \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") "
Nov 02 13:30:25.377576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.377515    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\" (UniqueName: \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\") pod \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\" (UID: \"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25\") "
Nov 02 13:30:25.377716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.377614    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\" (UniqueName: \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\") pod \"14a9f87a-1272-4591-bd5d-df59619b1f28\" (UID: \"14a9f87a-1272-4591-bd5d-df59619b1f28\") "
Nov 02 13:30:25.377884 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.377815    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume" pod "14a9f87a-1272-4591-bd5d-df59619b1f28" (UID: "14a9f87a-1272-4591-bd5d-df59619b1f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:25.378269 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.378202    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a9f87a-1272-4591-bd5d-df59619b1f28-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:25.378804 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.378778    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume" pod "54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" (UID: "54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:25.381385 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.381332    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh" pod "54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" (UID: "54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25"). InnerVolumeSpecName "kube-api-access-wwqjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:25.392882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.392853    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x" pod "14a9f87a-1272-4591-bd5d-df59619b1f28" (UID: "14a9f87a-1272-4591-bd5d-df59619b1f28"). InnerVolumeSpecName "kube-api-access-j556x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:25.479653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.479569    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j556x\" (UniqueName: \"kubernetes.io/projected/14a9f87a-1272-4591-bd5d-df59619b1f28-kube-api-access-j556x\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:25.479653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.479601    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:25.479653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.479613    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwqjh\" (UniqueName: \"kubernetes.io/projected/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25-kube-api-access-wwqjh\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:25.911602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:25.911561    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:26.250380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.250345    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"
Nov 02 13:30:26.250854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.250456    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr" event={"ID":"54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25","Type":"ContainerDied","Data":"aba31ac645fe418d2e16a876f3f79a2eacaeeb9d8cc1293fd297f2cf31496d8e"}
Nov 02 13:30:26.250854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.250523    4928 scope.go:122] "RemoveContainer" containerID="4882cca20dc362d6b02bc1ef2b7ca4ace978c233fafc9fc40cc79139c79bb27a"
Nov 02 13:30:26.250956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.250928    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"
Nov 02 13:30:26.308755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.308718    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 13:30:26.321521 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.321486    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843978-667d5f8bdc-pzvrr"]
Nov 02 13:30:26.349470 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.349436    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 13:30:26.354749 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.354718    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843978-575d4f4c49-zgpk7"]
Nov 02 13:30:26.497849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.497800    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" path="/var/lib/kubelet/pods/14a9f87a-1272-4591-bd5d-df59619b1f28/volumes"
Nov 02 13:30:26.498240 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.498226    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" path="/var/lib/kubelet/pods/54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25/volumes"
Nov 02 13:30:26.919016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:26.918979    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:29.192850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.191122    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 13:30:29.192850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.191468    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" podUID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" containerName="maas-api" containerID="cri-o://40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3" gracePeriod=30
Nov 02 13:30:29.528568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.528540    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 13:30:29.690848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.690222    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\") pod \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") "
Nov 02 13:30:29.690848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.690377    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\") pod \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") "
Nov 02 13:30:29.690848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.690440    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") pod \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") "
Nov 02 13:30:29.690848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.690514    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\" (UniqueName: \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\") pod \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\" (UID: \"39df5dfc-af28-4dcb-be2c-0e01a12c9519\") "
Nov 02 13:30:29.702886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.696435    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca" pod "39df5dfc-af28-4dcb-be2c-0e01a12c9519" (UID: "39df5dfc-af28-4dcb-be2c-0e01a12c9519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:29.702886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.701756    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m" pod "39df5dfc-af28-4dcb-be2c-0e01a12c9519" (UID: "39df5dfc-af28-4dcb-be2c-0e01a12c9519"). InnerVolumeSpecName "kube-api-access-psf6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:29.702886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.702461    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs" pod "39df5dfc-af28-4dcb-be2c-0e01a12c9519" (UID: "39df5dfc-af28-4dcb-be2c-0e01a12c9519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:29.703336 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.703311    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls" pod "39df5dfc-af28-4dcb-be2c-0e01a12c9519" (UID: "39df5dfc-af28-4dcb-be2c-0e01a12c9519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:29.792546 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.792456    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psf6m\" (UniqueName: \"kubernetes.io/projected/39df5dfc-af28-4dcb-be2c-0e01a12c9519-kube-api-access-psf6m\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:29.792546 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.792490    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:29.792546 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.792501    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39df5dfc-af28-4dcb-be2c-0e01a12c9519-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:29.792546 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:29.792509    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39df5dfc-af28-4dcb-be2c-0e01a12c9519-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:30.287912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.287877    4928 generic.go:356] "Generic (PLEG): container finished" podID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" containerID="40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3" exitCode=0
Nov 02 13:30:30.288376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.287925    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"
Nov 02 13:30:30.288376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.287938    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" event={"ID":"39df5dfc-af28-4dcb-be2c-0e01a12c9519","Type":"ContainerDied","Data":"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"}
Nov 02 13:30:30.288376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.287987    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d" event={"ID":"39df5dfc-af28-4dcb-be2c-0e01a12c9519","Type":"ContainerDied","Data":"4fc0946fd62f33da2d137306acbcad54d975f740e4c63c179113729ae088cb12"}
Nov 02 13:30:30.288376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.288009    4928 scope.go:122] "RemoveContainer" containerID="40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"
Nov 02 13:30:30.299186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.299166    4928 scope.go:122] "RemoveContainer" containerID="40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"
Nov 02 13:30:30.299471 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:30.299451    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3\": container with ID starting with 40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3 not found: ID does not exist" containerID="40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"
Nov 02 13:30:30.299541 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.299483    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3"} err="failed to get container status \"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3\": rpc error: code = NotFound desc = could not find container \"40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3\": container with ID starting with 40906c29e848691fedfc0eec21ad8d494dd2d5a7fb5303f8f38be199b195c2c3 not found: ID does not exist"
Nov 02 13:30:30.346004 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.345890    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 13:30:30.349753 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.349720    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e7561a-7789648446-lgr2d"]
Nov 02 13:30:30.498870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:30.498814    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" path="/var/lib/kubelet/pods/39df5dfc-af28-4dcb-be2c-0e01a12c9519/volumes"
Nov 02 13:30:35.911011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:35.910972    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:36.919427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:36.919389    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:39.730060 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:39.730019    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:30:40.270572 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.270532    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 13:30:40.270852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.270775    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerName="maas-api" containerID="cri-o://c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567" gracePeriod=30
Nov 02 13:30:40.344076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.344039    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.164:8443/health\": dial tcp 10.132.0.164:8443: connect: connection refused"
Nov 02 13:30:40.557548 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.557522    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 13:30:40.669660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.669591    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\" (UniqueName: \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\") pod \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") "
Nov 02 13:30:40.669811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.669798    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\") pod \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") "
Nov 02 13:30:40.669924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.669874    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\") pod \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") "
Nov 02 13:30:40.669972 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.669938    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\") pod \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\" (UID: \"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7\") "
Nov 02 13:30:40.670217 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.670183    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca" pod "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" (UID: "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:40.670566 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.670516    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:40.672052 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.672024    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls" pod "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" (UID: "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:40.672341 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.672317    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs" pod "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" (UID: "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:40.672472 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.672452    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj" pod "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" (UID: "77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7"). InnerVolumeSpecName "kube-api-access-648mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:40.772849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.772149    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648mj\" (UniqueName: \"kubernetes.io/projected/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-kube-api-access-648mj\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:40.772849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.772199    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:40.772849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:40.772231    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:41.391133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.391094    4928 generic.go:356] "Generic (PLEG): container finished" podID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerID="c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567" exitCode=0
Nov 02 13:30:41.391352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.391153    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"
Nov 02 13:30:41.391352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.391180    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" event={"ID":"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7","Type":"ContainerDied","Data":"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"}
Nov 02 13:30:41.391352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.391286    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4" event={"ID":"77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7","Type":"ContainerDied","Data":"cefa77d09f0c3b91b1c087adadb726907530ee1a2934c8d8717eb2fe55469e2a"}
Nov 02 13:30:41.391352 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.391325    4928 scope.go:122] "RemoveContainer" containerID="c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"
Nov 02 13:30:41.404686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.404667    4928 scope.go:122] "RemoveContainer" containerID="c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"
Nov 02 13:30:41.404959 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:41.404941    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567\": container with ID starting with c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567 not found: ID does not exist" containerID="c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"
Nov 02 13:30:41.405013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.404970    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567"} err="failed to get container status \"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567\": rpc error: code = NotFound desc = could not find container \"c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567\": container with ID starting with c930221e24b3befecd2d275fceca8325a47436bb7c7652086b64c249e5e3f567 not found: ID does not exist"
Nov 02 13:30:41.448207 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:41.448172    4928 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a6e6e4_6d96_4f01_9b2b_37a0952cc3a7.slice/crio-cefa77d09f0c3b91b1c087adadb726907530ee1a2934c8d8717eb2fe55469e2a\": RecentStats: unable to find data in memory cache]"
Nov 02 13:30:41.449899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.449876    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 13:30:41.463071 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:41.463040    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6416f0-6d4bfc6fc-sj5g4"]
Nov 02 13:30:42.077301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.077276    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 13:30:42.108434 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.108408    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 13:30:42.191043 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.191003    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\" (UniqueName: \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\") pod \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") "
Nov 02 13:30:42.191257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.191053    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\" (UniqueName: \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\") pod \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") "
Nov 02 13:30:42.191257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.191133    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\") pod \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\" (UID: \"11a385f5-ac60-4ee4-9c30-4356b4a9f913\") "
Nov 02 13:30:42.191421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.191277    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\") pod \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\" (UID: \"6504ebd1-aaa7-4bdc-8105-6b8991468c64\") "
Nov 02 13:30:42.192194 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.192165    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume" pod "6504ebd1-aaa7-4bdc-8105-6b8991468c64" (UID: "6504ebd1-aaa7-4bdc-8105-6b8991468c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:42.194863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.194822    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz" pod "11a385f5-ac60-4ee4-9c30-4356b4a9f913" (UID: "11a385f5-ac60-4ee4-9c30-4356b4a9f913"). InnerVolumeSpecName "kube-api-access-pznpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:42.195218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.195189    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume" pod "11a385f5-ac60-4ee4-9c30-4356b4a9f913" (UID: "11a385f5-ac60-4ee4-9c30-4356b4a9f913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:42.197476 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.197450    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp" pod "6504ebd1-aaa7-4bdc-8105-6b8991468c64" (UID: "6504ebd1-aaa7-4bdc-8105-6b8991468c64"). InnerVolumeSpecName "kube-api-access-5rmwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:42.292789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.292714    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6504ebd1-aaa7-4bdc-8105-6b8991468c64-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:42.292789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.292745    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pznpz\" (UniqueName: \"kubernetes.io/projected/11a385f5-ac60-4ee4-9c30-4356b4a9f913-kube-api-access-pznpz\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:42.292789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.292759    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rmwp\" (UniqueName: \"kubernetes.io/projected/6504ebd1-aaa7-4bdc-8105-6b8991468c64-kube-api-access-5rmwp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:42.292789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.292769    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a385f5-ac60-4ee4-9c30-4356b4a9f913-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:42.399433 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.399398    4928 generic.go:356] "Generic (PLEG): container finished" podID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerID="2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a" exitCode=0
Nov 02 13:30:42.399630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.399448    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"
Nov 02 13:30:42.399630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.399485    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" event={"ID":"6504ebd1-aaa7-4bdc-8105-6b8991468c64","Type":"ContainerDied","Data":"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"}
Nov 02 13:30:42.399630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.399536    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv" event={"ID":"6504ebd1-aaa7-4bdc-8105-6b8991468c64","Type":"ContainerDied","Data":"82ee612f88333471a729c8628aea2ea852532df8f4daee1f86eb37785affe609"}
Nov 02 13:30:42.399630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.399566    4928 scope.go:122] "RemoveContainer" containerID="2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"
Nov 02 13:30:42.401906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.401883    4928 generic.go:356] "Generic (PLEG): container finished" podID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerID="295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f" exitCode=0
Nov 02 13:30:42.402021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.401912    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" event={"ID":"11a385f5-ac60-4ee4-9c30-4356b4a9f913","Type":"ContainerDied","Data":"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"}
Nov 02 13:30:42.402021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.401926    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"
Nov 02 13:30:42.402021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.401965    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv" event={"ID":"11a385f5-ac60-4ee4-9c30-4356b4a9f913","Type":"ContainerDied","Data":"57a69cdcbbde2e8dcf1c21fe1357cf6a0a302bb8d77fc1d43694dce2faf10b36"}
Nov 02 13:30:42.412205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.412190    4928 scope.go:122] "RemoveContainer" containerID="2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"
Nov 02 13:30:42.412504 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:42.412478    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a\": container with ID starting with 2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a not found: ID does not exist" containerID="2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"
Nov 02 13:30:42.412580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.412519    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a"} err="failed to get container status \"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a\": rpc error: code = NotFound desc = could not find container \"2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a\": container with ID starting with 2994e6d2184f543349496d850d09691af288829c29ea3a5e9554ee6c405be44a not found: ID does not exist"
Nov 02 13:30:42.412580 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.412545    4928 scope.go:122] "RemoveContainer" containerID="295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"
Nov 02 13:30:42.422649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.422632    4928 scope.go:122] "RemoveContainer" containerID="295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"
Nov 02 13:30:42.422948 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:30:42.422918    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f\": container with ID starting with 295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f not found: ID does not exist" containerID="295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"
Nov 02 13:30:42.423018 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.422955    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f"} err="failed to get container status \"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f\": rpc error: code = NotFound desc = could not find container \"295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f\": container with ID starting with 295f18bf100f90e0c0f20840a24d5f85a8f236a4c9493507ddef5a8ec378f79f not found: ID does not exist"
Nov 02 13:30:42.498382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.498356    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" path="/var/lib/kubelet/pods/77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7/volumes"
Nov 02 13:30:42.506076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.506053    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 13:30:42.529333 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.529309    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e7561a-6b449b78d5-vv6lv"]
Nov 02 13:30:42.583160 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.583079    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 13:30:42.593105 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:42.593075    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e7561a-bb74f56f9-7mjlv"]
Nov 02 13:30:44.498419 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:44.498379    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" path="/var/lib/kubelet/pods/11a385f5-ac60-4ee4-9c30-4356b4a9f913/volumes"
Nov 02 13:30:44.498846 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:44.498817    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" path="/var/lib/kubelet/pods/6504ebd1-aaa7-4bdc-8105-6b8991468c64/volumes"
Nov 02 13:30:45.017351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:45.017322    4928 scope.go:122] "RemoveContainer" containerID="2044d0409ce35fb2ed2eb81112a1830203dc948b77652c6f91c4febb7b7f681c"
Nov 02 13:30:50.325979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:50.325582    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 13:30:50.325979 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:50.325923    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy" containerID="cri-o://eb8e5a6aca16c7a94cceef09e9a7b486519c02602d5f3d4870cf1ea19c52c518" gracePeriod=30
Nov 02 13:30:55.550444 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.549455    4928 generic.go:356] "Generic (PLEG): container finished" podID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerID="eb8e5a6aca16c7a94cceef09e9a7b486519c02602d5f3d4870cf1ea19c52c518" exitCode=0
Nov 02 13:30:55.550444 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.549522    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerDied","Data":"eb8e5a6aca16c7a94cceef09e9a7b486519c02602d5f3d4870cf1ea19c52c518"}
Nov 02 13:30:55.550444 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.549565    4928 scope.go:122] "RemoveContainer" containerID="37a2d5d63a6f63314960794fac2d631c5ce454117616c66d72f13d3f3fe29ee1"
Nov 02 13:30:55.618419 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.618388    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 13:30:55.677993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.677959    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678024    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678117    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678143    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678220    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678243    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678293    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678324    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:55.678498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678351    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678403    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\") pod \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\" (UID: \"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c\") "
Nov 02 13:30:55.678498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678404    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:55.678732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678585    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:55.678732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678681    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:55.678920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.678894    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:55.679124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.679100    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.679255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.679198    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.679255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.679215    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.679255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.679228    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.679255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.679243    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.681259 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.681236    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:30:55.681367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.681305    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "kube-api-access-2852h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:55.681367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.681341    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:55.685517 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.685488    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy" pod "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" (UID: "6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:55.781359 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.781319    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.781359 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.781356    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.781611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.781369    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2852h\" (UniqueName: \"kubernetes.io/projected/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-kube-api-access-2852h\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.781611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.781383    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.782484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.782463    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 13:30:55.789969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.789937    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 13:30:55.799344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:55.799312    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 13:30:56.558617 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:56.558580    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5" event={"ID":"6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c","Type":"ContainerDied","Data":"29d54a95fa4d2904a91b2bed938195cac1f3f6e954b1f93876158978f3bc84d6"}
Nov 02 13:30:56.558617 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:56.558604    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"
Nov 02 13:30:56.559204 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:56.558640    4928 scope.go:122] "RemoveContainer" containerID="eb8e5a6aca16c7a94cceef09e9a7b486519c02602d5f3d4870cf1ea19c52c518"
Nov 02 13:30:56.617482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:56.617450    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 13:30:56.630936 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:56.630904    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6416f0-openshift-default-6b984f4649-r6sn5"]
Nov 02 13:30:58.505484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:30:58.505445    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" path="/var/lib/kubelet/pods/6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c/volumes"
Nov 02 13:31:09.729294 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:31:09.729257    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:31:10.804764 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:10.804715    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main" containerID="cri-o://480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891" gracePeriod=45
Nov 02 13:31:10.811498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:10.811431    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" containerID="cri-o://b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791" gracePeriod=45
Nov 02 13:31:10.820109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:10.820056    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" containerID="cri-o://9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803" gracePeriod=45
Nov 02 13:31:11.021016 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.017200    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": dial tcp 10.132.0.75:8000: connect: connection refused"
Nov 02 13:31:11.151930 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.151602    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 13:31:11.174964 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.172023    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 13:31:11.238284 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.238250    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 13:31:11.287640 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287602    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.287844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287695    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.287844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287741    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.287844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287769    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.287844 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287794    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\" (UniqueName: \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.288088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287873    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.288088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287920    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.288088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287945    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.288088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.287964    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.288345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288115    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.288345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288121    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\" (UniqueName: \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\") pod \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\" (UID: \"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5\") "
Nov 02 13:31:11.288345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288170    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.288345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288218    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\" (UniqueName: \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.288345 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288259    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\") pod \"f41d1b96-094b-4bc1-ad11-684ca5244607\" (UID: \"f41d1b96-094b-4bc1-ad11-684ca5244607\") "
Nov 02 13:31:11.288613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288484    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.288740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288716    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.288810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288764    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.288810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288782    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.288810 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288796    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.289008 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.288917    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.289670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.289642    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.290519 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.290474    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:11.291542 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.291515    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:11.292340 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.292320    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.297200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.297172    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.297325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.297199    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6" pod "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" (UID: "d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5"). InnerVolumeSpecName "kube-api-access-q74w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:11.298450 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.298427    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz" pod "f41d1b96-094b-4bc1-ad11-684ca5244607" (UID: "f41d1b96-094b-4bc1-ad11-684ca5244607"). InnerVolumeSpecName "kube-api-access-rjkcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:11.390218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390131    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\" (UniqueName: \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390249    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390289    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390314    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390402 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390357    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390414    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\") pod \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\" (UID: \"d6357a7d-7a1a-4a03-a008-e0e97b6413d1\") "
Nov 02 13:31:11.390632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390572    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.390632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390583    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.390792 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.390757    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.391109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391063    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391090    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkcz\" (UniqueName: \"kubernetes.io/projected/f41d1b96-094b-4bc1-ad11-684ca5244607-kube-api-access-rjkcz\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391107    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391122    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391136    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391147    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391160    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391173    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q74w6\" (UniqueName: \"kubernetes.io/projected/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5-kube-api-access-q74w6\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391185    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391198    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391209    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41d1b96-094b-4bc1-ad11-684ca5244607-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.391344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.391220    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41d1b96-094b-4bc1-ad11-684ca5244607-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.392623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.392601    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "kube-api-access-c6bsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:11.393026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.392999    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.393124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.393020    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs" pod "d6357a7d-7a1a-4a03-a008-e0e97b6413d1" (UID: "d6357a7d-7a1a-4a03-a008-e0e97b6413d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:11.491781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.491750    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.491781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.491778    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.491781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.491787    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6bsf\" (UniqueName: \"kubernetes.io/projected/d6357a7d-7a1a-4a03-a008-e0e97b6413d1-kube-api-access-c6bsf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.695798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.695707    4928 generic.go:356] "Generic (PLEG): container finished" podID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerID="9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803" exitCode=0
Nov 02 13:31:11.695798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.695753    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"
Nov 02 13:31:11.696078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.695796    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" event={"ID":"d6357a7d-7a1a-4a03-a008-e0e97b6413d1","Type":"ContainerDied","Data":"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"}
Nov 02 13:31:11.696078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.695865    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw" event={"ID":"d6357a7d-7a1a-4a03-a008-e0e97b6413d1","Type":"ContainerDied","Data":"d874bf7739f7545c7417d70e1eb67b03dd842f9fd5e1aba3223e50458b3eac00"}
Nov 02 13:31:11.696078 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.695891    4928 scope.go:122] "RemoveContainer" containerID="9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"
Nov 02 13:31:11.697311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.697289    4928 generic.go:356] "Generic (PLEG): container finished" podID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerID="b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791" exitCode=0
Nov 02 13:31:11.697421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.697320    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" event={"ID":"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5","Type":"ContainerDied","Data":"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"}
Nov 02 13:31:11.697421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.697336    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"
Nov 02 13:31:11.697421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.697355    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" event={"ID":"d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5","Type":"ContainerDied","Data":"f6432d00165f2b28e2e4f58490a6a8299fa2ba357dbd2729e5deeb4f08b0f8d6"}
Nov 02 13:31:11.699011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.698976    4928 generic.go:356] "Generic (PLEG): container finished" podID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerID="480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891" exitCode=0
Nov 02 13:31:11.699120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.699018    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"
Nov 02 13:31:11.699120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.699037    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" event={"ID":"f41d1b96-094b-4bc1-ad11-684ca5244607","Type":"ContainerDied","Data":"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"}
Nov 02 13:31:11.699120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.699062    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk" event={"ID":"f41d1b96-094b-4bc1-ad11-684ca5244607","Type":"ContainerDied","Data":"a2abffbadd70c4167fcf2b136aa30cb6ba89759d6cc51e4de6838f8aece55c34"}
Nov 02 13:31:11.708525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.708500    4928 scope.go:122] "RemoveContainer" containerID="9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"
Nov 02 13:31:11.708906 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:31:11.708802    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803\": container with ID starting with 9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803 not found: ID does not exist" containerID="9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"
Nov 02 13:31:11.708906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.708869    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803"} err="failed to get container status \"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803\": rpc error: code = NotFound desc = could not find container \"9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803\": container with ID starting with 9f37184c6f981b80762a865c059f42051da544663532dd6934bb62874dbf6803 not found: ID does not exist"
Nov 02 13:31:11.708906 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.708887    4928 scope.go:122] "RemoveContainer" containerID="b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"
Nov 02 13:31:11.722118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.722096    4928 scope.go:122] "RemoveContainer" containerID="b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"
Nov 02 13:31:11.722448 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:31:11.722426    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791\": container with ID starting with b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791 not found: ID does not exist" containerID="b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"
Nov 02 13:31:11.722528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.722461    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791"} err="failed to get container status \"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791\": rpc error: code = NotFound desc = could not find container \"b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791\": container with ID starting with b12712a4643ab872ee3a624549e6582c0e22a0a197e8271583e89df11c001791 not found: ID does not exist"
Nov 02 13:31:11.722528 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.722484    4928 scope.go:122] "RemoveContainer" containerID="480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"
Nov 02 13:31:11.733478 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.733459    4928 scope.go:122] "RemoveContainer" containerID="480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"
Nov 02 13:31:11.733772 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:31:11.733747    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891\": container with ID starting with 480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891 not found: ID does not exist" containerID="480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"
Nov 02 13:31:11.733851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.733782    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891"} err="failed to get container status \"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891\": rpc error: code = NotFound desc = could not find container \"480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891\": container with ID starting with 480d77e08d8efe02636314c5e6c4d4b481ab3b01116548e13f5fc99a6ebf6891 not found: ID does not exist"
Nov 02 13:31:11.752166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.752134    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 13:31:11.755045 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.755023    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb"]
Nov 02 13:31:11.797722 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.797689    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 13:31:11.801301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.801273    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/distinct-2-w3-068756-kserve-756647c76f-4fsmk"]
Nov 02 13:31:11.838291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.838259    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 13:31:11.840997 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:11.840975    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/premium-w3-068756-kserve-7f75685cc9-hqpkw"]
Nov 02 13:31:12.130336 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:12.130291    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/distinct-w3-068756-kserve-6598b674d6-cfghb" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:31:12.498369 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:12.498334    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" path="/var/lib/kubelet/pods/d6357a7d-7a1a-4a03-a008-e0e97b6413d1/volumes"
Nov 02 13:31:12.498806 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:12.498791    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" path="/var/lib/kubelet/pods/d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5/volumes"
Nov 02 13:31:12.499262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:12.499244    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" path="/var/lib/kubelet/pods/f41d1b96-094b-4bc1-ad11-684ca5244607/volumes"
Nov 02 13:31:39.741039 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:31:39.740993    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:31:39.754420 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:39.754385    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 13:31:39.754679 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:39.754651    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" podUID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" containerName="maas-api" containerID="cri-o://693236a15e400921a03bd3a76d9c52859e682f6da2ce6096f579e402386e42ee" gracePeriod=30
Nov 02 13:31:40.002902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:39.998777    4928 generic.go:356] "Generic (PLEG): container finished" podID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" containerID="693236a15e400921a03bd3a76d9c52859e682f6da2ce6096f579e402386e42ee" exitCode=0
Nov 02 13:31:40.025858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.013216    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" event={"ID":"cdcaa75b-7486-498d-80eb-97324ac2bdc2","Type":"ContainerDied","Data":"693236a15e400921a03bd3a76d9c52859e682f6da2ce6096f579e402386e42ee"}
Nov 02 13:31:40.068897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.068870    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 13:31:40.091602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.091552    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\" (UniqueName: \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\") pod \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") "
Nov 02 13:31:40.091795 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.091625    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\") pod \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") "
Nov 02 13:31:40.091795 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.091708    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\") pod \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") "
Nov 02 13:31:40.091970 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.091794    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\") pod \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\" (UID: \"cdcaa75b-7486-498d-80eb-97324ac2bdc2\") "
Nov 02 13:31:40.092483 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.092442    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca" pod "cdcaa75b-7486-498d-80eb-97324ac2bdc2" (UID: "cdcaa75b-7486-498d-80eb-97324ac2bdc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:40.160849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.140463    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl" pod "cdcaa75b-7486-498d-80eb-97324ac2bdc2" (UID: "cdcaa75b-7486-498d-80eb-97324ac2bdc2"). InnerVolumeSpecName "kube-api-access-zldpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:40.160849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.140550    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls" pod "cdcaa75b-7486-498d-80eb-97324ac2bdc2" (UID: "cdcaa75b-7486-498d-80eb-97324ac2bdc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:40.160849 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.141024    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs" pod "cdcaa75b-7486-498d-80eb-97324ac2bdc2" (UID: "cdcaa75b-7486-498d-80eb-97324ac2bdc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:40.194850 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.194612    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcaa75b-7486-498d-80eb-97324ac2bdc2-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:40.218120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.194671    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:40.218120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.201940    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zldpl\" (UniqueName: \"kubernetes.io/projected/cdcaa75b-7486-498d-80eb-97324ac2bdc2-kube-api-access-zldpl\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:40.218120 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:40.201967    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcaa75b-7486-498d-80eb-97324ac2bdc2-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:31:41.022315 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:41.022284    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"
Nov 02 13:31:41.022882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:41.022285    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g" event={"ID":"cdcaa75b-7486-498d-80eb-97324ac2bdc2","Type":"ContainerDied","Data":"05bf521b65feb5cfd6372d23195dbe1bf3df0179735b0d1e0c5f1ebe4523e7c7"}
Nov 02 13:31:41.022882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:41.022429    4928 scope.go:122] "RemoveContainer" containerID="693236a15e400921a03bd3a76d9c52859e682f6da2ce6096f579e402386e42ee"
Nov 02 13:31:41.079637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:41.079604    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 13:31:41.093329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:41.093296    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-068756-dccdd974-ctf7g"]
Nov 02 13:31:42.497701 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:42.497654    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" path="/var/lib/kubelet/pods/cdcaa75b-7486-498d-80eb-97324ac2bdc2/volumes"
Nov 02 13:31:43.358504 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:43.358469    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:31:43.374227 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:31:43.374203    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:32:09.729931 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:32:09.729886    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:32:39.806588 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:32:39.806507    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:33:09.729769 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:09.729724    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:33:24.585929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:24.585820    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 13:33:26.047253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.047216    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 13:33:26.047721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.047443    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" podUID="91ed7422-66b4-4198-9b85-956064954579" containerName="maas-api" containerID="cri-o://e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30" gracePeriod=30
Nov 02 13:33:26.199929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.199883    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" podUID="91ed7422-66b4-4198-9b85-956064954579" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.153:8443/health\": dial tcp 10.132.0.153:8443: connect: connection refused"
Nov 02 13:33:26.463203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.463166    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 13:33:26.669033 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.668932    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\" (UniqueName: \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\") pod \"91ed7422-66b4-4198-9b85-956064954579\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") "
Nov 02 13:33:26.669214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.669062    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\") pod \"91ed7422-66b4-4198-9b85-956064954579\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") "
Nov 02 13:33:26.669214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.669126    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\") pod \"91ed7422-66b4-4198-9b85-956064954579\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") "
Nov 02 13:33:26.669214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.669177    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\") pod \"91ed7422-66b4-4198-9b85-956064954579\" (UID: \"91ed7422-66b4-4198-9b85-956064954579\") "
Nov 02 13:33:26.669789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.669746    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca" pod "91ed7422-66b4-4198-9b85-956064954579" (UID: "91ed7422-66b4-4198-9b85-956064954579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:26.671361 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.671335    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7" pod "91ed7422-66b4-4198-9b85-956064954579" (UID: "91ed7422-66b4-4198-9b85-956064954579"). InnerVolumeSpecName "kube-api-access-g8kr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:26.671510 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.671480    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls" pod "91ed7422-66b4-4198-9b85-956064954579" (UID: "91ed7422-66b4-4198-9b85-956064954579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:26.671615 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.671538    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs" pod "91ed7422-66b4-4198-9b85-956064954579" (UID: "91ed7422-66b4-4198-9b85-956064954579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:26.770377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.770337    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:26.770377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.770370    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8kr7\" (UniqueName: \"kubernetes.io/projected/91ed7422-66b4-4198-9b85-956064954579-kube-api-access-g8kr7\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:26.770377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.770380    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed7422-66b4-4198-9b85-956064954579-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:26.770685 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.770389    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed7422-66b4-4198-9b85-956064954579-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:26.915312 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.915269    4928 generic.go:356] "Generic (PLEG): container finished" podID="91ed7422-66b4-4198-9b85-956064954579" containerID="e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30" exitCode=0
Nov 02 13:33:26.915537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.915328    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"
Nov 02 13:33:26.915537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.915352    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" event={"ID":"91ed7422-66b4-4198-9b85-956064954579","Type":"ContainerDied","Data":"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"}
Nov 02 13:33:26.915537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.915412    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz" event={"ID":"91ed7422-66b4-4198-9b85-956064954579","Type":"ContainerDied","Data":"447b84793320b5ba29e29bd93e67ccd5d773aa2e9518b18a294c522d4ce15139"}
Nov 02 13:33:26.915537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.915437    4928 scope.go:122] "RemoveContainer" containerID="e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"
Nov 02 13:33:26.932149 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.932092    4928 scope.go:122] "RemoveContainer" containerID="e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"
Nov 02 13:33:26.933075 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:26.933034    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30\": container with ID starting with e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30 not found: ID does not exist" containerID="e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"
Nov 02 13:33:26.933187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.933078    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30"} err="failed to get container status \"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30\": rpc error: code = NotFound desc = could not find container \"e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30\": container with ID starting with e7ce6c840a5d0b3fc231ba6d7dbb945a7e474864f96d428abfe0ac7c8048ca30 not found: ID does not exist"
Nov 02 13:33:26.981699 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.981665    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 13:33:26.986231 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:26.986200    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a6f2c0-cd799b7fc-bd9wz"]
Nov 02 13:33:28.497978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:28.497942    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ed7422-66b4-4198-9b85-956064954579" path="/var/lib/kubelet/pods/91ed7422-66b4-4198-9b85-956064954579/volumes"
Nov 02 13:33:29.586983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:29.586937    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing" containerID="cri-o://0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f" gracePeriod=25
Nov 02 13:33:31.854702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:31.854657    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:34.215665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:34.215626    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 13:33:34.216177 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:34.215901    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy" containerID="cri-o://9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6" gracePeriod=30
Nov 02 13:33:39.507207 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.507179    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 13:33:39.649855 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.649803    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.649895    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.649965    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.649997    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650045    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650143    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650139    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:39.650368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650189    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650234    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650264    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\") pod \"dbee72f1-e429-4ca4-bbf5-076da013775f\" (UID: \"dbee72f1-e429-4ca4-bbf5-076da013775f\") "
Nov 02 13:33:39.650630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650410    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:39.650630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650432    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:39.650630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650447    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:39.650787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650729    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:39.650920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650870    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.650920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650894    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.650920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650907    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbee72f1-e429-4ca4-bbf5-076da013775f-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.650920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650920    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.651182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.650933    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.652417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.652383    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:33:39.652956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.652932    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:39.653063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.652953    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:39.653063 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.652997    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt" pod "dbee72f1-e429-4ca4-bbf5-076da013775f" (UID: "dbee72f1-e429-4ca4-bbf5-076da013775f"). InnerVolumeSpecName "kube-api-access-mx4kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:39.729275 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:39.729238    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:33:39.752637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.752561    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.752637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.752599    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.752637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.752614    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbee72f1-e429-4ca4-bbf5-076da013775f-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:39.752637 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:39.752630    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4kt\" (UniqueName: \"kubernetes.io/projected/dbee72f1-e429-4ca4-bbf5-076da013775f-kube-api-access-mx4kt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:40.016456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.016375    4928 generic.go:356] "Generic (PLEG): container finished" podID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerID="9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6" exitCode=0
Nov 02 13:33:40.016456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.016426    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"
Nov 02 13:33:40.016663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.016445    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" event={"ID":"dbee72f1-e429-4ca4-bbf5-076da013775f","Type":"ContainerDied","Data":"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"}
Nov 02 13:33:40.016663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.016505    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s" event={"ID":"dbee72f1-e429-4ca4-bbf5-076da013775f","Type":"ContainerDied","Data":"511c31b4722ca46aaef869716c261d08cfad0899a503c6859c3d07aa8dd8f65c"}
Nov 02 13:33:40.016663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.016533    4928 scope.go:122] "RemoveContainer" containerID="9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"
Nov 02 13:33:40.028257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.028232    4928 scope.go:122] "RemoveContainer" containerID="9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"
Nov 02 13:33:40.028578 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:40.028552    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6\": container with ID starting with 9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6 not found: ID does not exist" containerID="9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"
Nov 02 13:33:40.028648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.028590    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6"} err="failed to get container status \"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6\": rpc error: code = NotFound desc = could not find container \"9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6\": container with ID starting with 9fa8f76faa970c9a39ec8fcff769941ab8f2ee56204f3238ee86ed5a5de162d6 not found: ID does not exist"
Nov 02 13:33:40.076887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.076853    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 13:33:40.081383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.081355    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-ddl4s"]
Nov 02 13:33:40.497552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:40.497515    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" path="/var/lib/kubelet/pods/dbee72f1-e429-4ca4-bbf5-076da013775f/volumes"
Nov 02 13:33:41.855049 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:41.855009    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:46.353044 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.352999    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:33:46.353576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.353285    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" podUID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" containerName="maas-api" containerID="cri-o://d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5" gracePeriod=30
Nov 02 13:33:46.689108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.689078    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:33:46.870890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.870846    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\") pod \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") "
Nov 02 13:33:46.871088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.870985    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\" (UniqueName: \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\") pod \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") "
Nov 02 13:33:46.871088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.871018    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\") pod \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") "
Nov 02 13:33:46.871088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.871054    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\") pod \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\" (UID: \"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1\") "
Nov 02 13:33:46.871510 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.871476    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca" pod "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" (UID: "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:46.873200 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.873172    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls" pod "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" (UID: "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:46.873417 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.873391    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q" pod "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" (UID: "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1"). InnerVolumeSpecName "kube-api-access-2sb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:46.873905 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.873883    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs" pod "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" (UID: "ba0cedb3-905a-4ba2-ad3c-b751ef431ec1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:46.972286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.972206    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:46.972286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.972234    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb2q\" (UniqueName: \"kubernetes.io/projected/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-kube-api-access-2sb2q\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:46.972286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.972243    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:46.972286 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:46.972255    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:47.098388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.098092    4928 generic.go:356] "Generic (PLEG): container finished" podID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" containerID="d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5" exitCode=0
Nov 02 13:33:47.100857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.098268    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" event={"ID":"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1","Type":"ContainerDied","Data":"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"}
Nov 02 13:33:47.100857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.099454    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x" event={"ID":"ba0cedb3-905a-4ba2-ad3c-b751ef431ec1","Type":"ContainerDied","Data":"db07f1ed31b221bbaf183f22bc347d9ad15314bd2175c0861f8cbc3d66490e73"}
Nov 02 13:33:47.100857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.099498    4928 scope.go:122] "RemoveContainer" containerID="d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"
Nov 02 13:33:47.100857 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.099718    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"
Nov 02 13:33:47.116354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.116335    4928 scope.go:122] "RemoveContainer" containerID="d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"
Nov 02 13:33:47.116673 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:47.116648    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5\": container with ID starting with d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5 not found: ID does not exist" containerID="d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"
Nov 02 13:33:47.116762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.116681    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5"} err="failed to get container status \"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5\": rpc error: code = NotFound desc = could not find container \"d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5\": container with ID starting with d36848a227a4141ca2b1e1d9cd57c671464a45be43867bcfd9753987ab4880c5 not found: ID does not exist"
Nov 02 13:33:47.160670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.160636    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:33:47.163672 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:47.163647    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e422b-78bf44cc8b-5k29x"]
Nov 02 13:33:48.497840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:48.497783    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" path="/var/lib/kubelet/pods/ba0cedb3-905a-4ba2-ad3c-b751ef431ec1/volumes"
Nov 02 13:33:49.863643 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.863615    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 13:33:49.912389 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.912354    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\") pod \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") "
Nov 02 13:33:49.912576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.912452    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\" (UniqueName: \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\") pod \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\" (UID: \"a74e4841-d2ba-4a0d-82ab-a047e71ac45d\") "
Nov 02 13:33:49.912780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.912755    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume" pod "a74e4841-d2ba-4a0d-82ab-a047e71ac45d" (UID: "a74e4841-d2ba-4a0d-82ab-a047e71ac45d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:49.913077 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.913057    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:49.916928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:49.916899    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp" pod "a74e4841-d2ba-4a0d-82ab-a047e71ac45d" (UID: "a74e4841-d2ba-4a0d-82ab-a047e71ac45d"). InnerVolumeSpecName "kube-api-access-km6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:50.013746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.013710    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6pp\" (UniqueName: \"kubernetes.io/projected/a74e4841-d2ba-4a0d-82ab-a047e71ac45d-kube-api-access-km6pp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:50.128357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.126892    4928 generic.go:356] "Generic (PLEG): container finished" podID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerID="0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f" exitCode=0
Nov 02 13:33:50.128357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.127111    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" event={"ID":"a74e4841-d2ba-4a0d-82ab-a047e71ac45d","Type":"ContainerDied","Data":"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"}
Nov 02 13:33:50.128357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.127164    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72" event={"ID":"a74e4841-d2ba-4a0d-82ab-a047e71ac45d","Type":"ContainerDied","Data":"7062b4c37761ea67def5bc6dae2ced4e543289d4fb09d4734416f8d0c9b77a5c"}
Nov 02 13:33:50.128357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.127192    4928 scope.go:122] "RemoveContainer" containerID="0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"
Nov 02 13:33:50.128357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.127303    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"
Nov 02 13:33:50.139277 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.139257    4928 scope.go:122] "RemoveContainer" containerID="0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"
Nov 02 13:33:50.139542 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:50.139526    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f\": container with ID starting with 0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f not found: ID does not exist" containerID="0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"
Nov 02 13:33:50.139586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.139550    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f"} err="failed to get container status \"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f\": rpc error: code = NotFound desc = could not find container \"0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f\": container with ID starting with 0b58730d7a92d70733aa4952700abea2a72550024d4360f851445651bbaf181f not found: ID does not exist"
Nov 02 13:33:50.183036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.183007    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 13:33:50.189664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.189632    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a6f2c0-6fd44bffcf-knx72"]
Nov 02 13:33:50.497918 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:50.497887    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" path="/var/lib/kubelet/pods/a74e4841-d2ba-4a0d-82ab-a047e71ac45d/volumes"
Nov 02 13:33:51.291383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.291346    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 13:33:51.291955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.291601    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" podUID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" containerName="maas-api" containerID="cri-o://45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b" gracePeriod=30
Nov 02 13:33:51.616697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.616661    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 13:33:51.749729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.747282    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\" (UniqueName: \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\") pod \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") "
Nov 02 13:33:51.749729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.747363    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\") pod \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") "
Nov 02 13:33:51.749729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.747494    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\") pod \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") "
Nov 02 13:33:51.749729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.747572    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\") pod \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\" (UID: \"1bdc46b5-2b8e-467f-b888-b269c3bbba6f\") "
Nov 02 13:33:51.749729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.748378    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca" pod "1bdc46b5-2b8e-467f-b888-b269c3bbba6f" (UID: "1bdc46b5-2b8e-467f-b888-b269c3bbba6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:51.756358 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.756321    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr" pod "1bdc46b5-2b8e-467f-b888-b269c3bbba6f" (UID: "1bdc46b5-2b8e-467f-b888-b269c3bbba6f"). InnerVolumeSpecName "kube-api-access-bm4mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:51.757156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.757130    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls" pod "1bdc46b5-2b8e-467f-b888-b269c3bbba6f" (UID: "1bdc46b5-2b8e-467f-b888-b269c3bbba6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:51.757533 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.757509    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs" pod "1bdc46b5-2b8e-467f-b888-b269c3bbba6f" (UID: "1bdc46b5-2b8e-467f-b888-b269c3bbba6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:51.848914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.848822    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4mr\" (UniqueName: \"kubernetes.io/projected/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-kube-api-access-bm4mr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:51.848914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.848871    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:51.848914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.848881    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:51.848914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.848889    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc46b5-2b8e-467f-b888-b269c3bbba6f-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:51.935013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.932728    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:33:51.935013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:51.933016    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy" containerID="cri-o://c9b31636092d66928c1332b5168dbcab5255caf712d54757dbace4b046e5c8bc" gracePeriod=30
Nov 02 13:33:52.153907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.152025    4928 generic.go:356] "Generic (PLEG): container finished" podID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" containerID="45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b" exitCode=0
Nov 02 13:33:52.153907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.152213    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" event={"ID":"1bdc46b5-2b8e-467f-b888-b269c3bbba6f","Type":"ContainerDied","Data":"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"}
Nov 02 13:33:52.153907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.152267    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2" event={"ID":"1bdc46b5-2b8e-467f-b888-b269c3bbba6f","Type":"ContainerDied","Data":"e3be77491727509efff06f5b6aded42e7b5d81f6f1db1dc7eaafc1ee9b78e120"}
Nov 02 13:33:52.153907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.152297    4928 scope.go:122] "RemoveContainer" containerID="45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"
Nov 02 13:33:52.153907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.152347    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"
Nov 02 13:33:52.170382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.170359    4928 scope.go:122] "RemoveContainer" containerID="45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"
Nov 02 13:33:52.170756 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:33:52.170728    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b\": container with ID starting with 45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b not found: ID does not exist" containerID="45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"
Nov 02 13:33:52.170852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.170766    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b"} err="failed to get container status \"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b\": rpc error: code = NotFound desc = could not find container \"45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b\": container with ID starting with 45108e480eecc81034b056192a449555e9746e78f6cb03b9d4ba2c85caac0e5b not found: ID does not exist"
Nov 02 13:33:52.215729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.215697    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 13:33:52.219594 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.219566    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1aa18c-8484db86c7-kkqr2"]
Nov 02 13:33:52.496840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:52.496796    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" path="/var/lib/kubelet/pods/1bdc46b5-2b8e-467f-b888-b269c3bbba6f/volumes"
Nov 02 13:33:57.188798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.188766    4928 generic.go:356] "Generic (PLEG): container finished" podID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerID="c9b31636092d66928c1332b5168dbcab5255caf712d54757dbace4b046e5c8bc" exitCode=0
Nov 02 13:33:57.189360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.188931    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerDied","Data":"c9b31636092d66928c1332b5168dbcab5255caf712d54757dbace4b046e5c8bc"}
Nov 02 13:33:57.189360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.189019    4928 scope.go:122] "RemoveContainer" containerID="d618487c66deff3093ffb75e7c9932be319202e9e0ae11309ddda238e898aa49"
Nov 02 13:33:57.217553 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.217531    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:33:57.306061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306022    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306150    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306184    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306215    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306439 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306239    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306439 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306385    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306450    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306495    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.306552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.306540    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\") pod \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\" (UID: \"34d55fcd-fa53-452a-bb7e-1134b9581b6a\") "
Nov 02 13:33:57.307035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.307005    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:57.307185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.307158    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:57.307358 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.307331    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:57.307470 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.307440    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:57.307633 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.307610    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:57.309003 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.308972    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:33:57.309305 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.309277    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "kube-api-access-7nllw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:57.309663 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.309643    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:57.309724 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.309668    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token" pod "34d55fcd-fa53-452a-bb7e-1134b9581b6a" (UID: "34d55fcd-fa53-452a-bb7e-1134b9581b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416040    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nllw\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-kube-api-access-7nllw\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416085    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416103    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416118    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416134    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416148    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416162    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d55fcd-fa53-452a-bb7e-1134b9581b6a-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416175    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:57.416273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:57.416188    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d55fcd-fa53-452a-bb7e-1134b9581b6a-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:33:58.136746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.136709    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 13:33:58.204696 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.204653    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7" event={"ID":"34d55fcd-fa53-452a-bb7e-1134b9581b6a","Type":"ContainerDied","Data":"a5e98587ad931af053707795cb656f51467c14f8da2cce4e0d90535181c9fc9e"}
Nov 02 13:33:58.205205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.204699    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"
Nov 02 13:33:58.205205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.204741    4928 scope.go:122] "RemoveContainer" containerID="c9b31636092d66928c1332b5168dbcab5255caf712d54757dbace4b046e5c8bc"
Nov 02 13:33:58.294885 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.294850    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:33:58.299597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.299571    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e422b-openshift-default-8577f86657-c8pb7"]
Nov 02 13:33:58.497862 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:33:58.497798    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" path="/var/lib/kubelet/pods/34d55fcd-fa53-452a-bb7e-1134b9581b6a/volumes"
Nov 02 13:34:01.717562 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:01.717519    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 13:34:03.137325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:03.137262    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing" containerID="cri-o://3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814" gracePeriod=25
Nov 02 13:34:04.902203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:04.902159    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:09.268212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:09.268169    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 13:34:09.300181 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:09.300145    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 13:34:09.729816 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:09.729722    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:34:14.268692 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:14.268649    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing" containerID="cri-o://1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee" gracePeriod=25
Nov 02 13:34:14.300983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:14.300917    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing" containerID="cri-o://30b7132eb185b3984a96b355d3131c14bb8f9e6a8442893f6909f99c247ef07d" gracePeriod=25
Nov 02 13:34:14.918312 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:14.917508    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:16.738339 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:16.738291    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main" containerID="cri-o://b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87" gracePeriod=45
Nov 02 13:34:17.044293 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.044259    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 13:34:17.104182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104151    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.104458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104238    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.104458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104364    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.104458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104435    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.105306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104474    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.105306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104508    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\" (UniqueName: \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\") pod \"9206a4db-80cd-45f5-b047-b130365277eb\" (UID: \"9206a4db-80cd-45f5-b047-b130365277eb\") "
Nov 02 13:34:17.105306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.104651    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.105306 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.105279    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.105568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.105373    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.105568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.105085    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.109974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.107805    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:17.109974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.109128    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.109974 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.109222    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j" pod "9206a4db-80cd-45f5-b047-b130365277eb" (UID: "9206a4db-80cd-45f5-b047-b130365277eb"). InnerVolumeSpecName "kube-api-access-n867j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:17.207125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.207094    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206a4db-80cd-45f5-b047-b130365277eb-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.207125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.207121    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.207125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.207129    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.207125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.207139    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206a4db-80cd-45f5-b047-b130365277eb-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.207517 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.207150    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n867j\" (UniqueName: \"kubernetes.io/projected/9206a4db-80cd-45f5-b047-b130365277eb-kube-api-access-n867j\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.374096 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.373991    4928 generic.go:356] "Generic (PLEG): container finished" podID="9206a4db-80cd-45f5-b047-b130365277eb" containerID="b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87" exitCode=0
Nov 02 13:34:17.374096 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.374041    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" event={"ID":"9206a4db-80cd-45f5-b047-b130365277eb","Type":"ContainerDied","Data":"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"}
Nov 02 13:34:17.374374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.374105    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj" event={"ID":"9206a4db-80cd-45f5-b047-b130365277eb","Type":"ContainerDied","Data":"040fadbdc31575a31a76a76a976c7bb2384a80d32db8e0e0a7f01bae1ce195c4"}
Nov 02 13:34:17.374374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.374129    4928 scope.go:122] "RemoveContainer" containerID="b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"
Nov 02 13:34:17.374374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.374054    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"
Nov 02 13:34:17.386154 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.386131    4928 scope.go:122] "RemoveContainer" containerID="b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"
Nov 02 13:34:17.386504 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:17.386484    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87\": container with ID starting with b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87 not found: ID does not exist" containerID="b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"
Nov 02 13:34:17.386586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.386514    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87"} err="failed to get container status \"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87\": rpc error: code = NotFound desc = could not find container \"b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87\": container with ID starting with b4e0905a2178aaf2a871af11772ec11a3336b3160ff37a23c26b2d19b2cd6d87 not found: ID does not exist"
Nov 02 13:34:17.434840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.434803    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 13:34:17.450781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:17.450744    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/simulator-w1-61e341-kserve-6c9fb87686-mcbqj"]
Nov 02 13:34:18.497243 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:18.497207    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9206a4db-80cd-45f5-b047-b130365277eb" path="/var/lib/kubelet/pods/9206a4db-80cd-45f5-b047-b130365277eb/volumes"
Nov 02 13:34:19.469858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:19.469791    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:19.469858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:19.469813    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:20.121235 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.121196    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 13:34:20.121754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.121430    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" podUID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" containerName="maas-api" containerID="cri-o://ea4d12d3d8e41b83915c421711f2ec30e62a6e9507deae0b568c0f7001e38b5c" gracePeriod=30
Nov 02 13:34:20.418282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.418034    4928 generic.go:356] "Generic (PLEG): container finished" podID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" containerID="ea4d12d3d8e41b83915c421711f2ec30e62a6e9507deae0b568c0f7001e38b5c" exitCode=0
Nov 02 13:34:20.418282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.418164    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" event={"ID":"b0fcc9b0-cf30-4e1a-99b7-b289f329e832","Type":"ContainerDied","Data":"ea4d12d3d8e41b83915c421711f2ec30e62a6e9507deae0b568c0f7001e38b5c"}
Nov 02 13:34:20.447958 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.447811    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 13:34:20.559290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.558240    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\" (UniqueName: \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\") pod \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") "
Nov 02 13:34:20.559290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.558345    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\") pod \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") "
Nov 02 13:34:20.559290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.558404    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\") pod \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") "
Nov 02 13:34:20.559290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.558484    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\") pod \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\" (UID: \"b0fcc9b0-cf30-4e1a-99b7-b289f329e832\") "
Nov 02 13:34:20.561290 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.560406    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca" pod "b0fcc9b0-cf30-4e1a-99b7-b289f329e832" (UID: "b0fcc9b0-cf30-4e1a-99b7-b289f329e832"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:20.573864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.572931    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp" pod "b0fcc9b0-cf30-4e1a-99b7-b289f329e832" (UID: "b0fcc9b0-cf30-4e1a-99b7-b289f329e832"). InnerVolumeSpecName "kube-api-access-qxjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:20.573864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.573100    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs" pod "b0fcc9b0-cf30-4e1a-99b7-b289f329e832" (UID: "b0fcc9b0-cf30-4e1a-99b7-b289f329e832"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:20.573864 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.573115    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls" pod "b0fcc9b0-cf30-4e1a-99b7-b289f329e832" (UID: "b0fcc9b0-cf30-4e1a-99b7-b289f329e832"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:20.661062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.661020    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxjkp\" (UniqueName: \"kubernetes.io/projected/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-kube-api-access-qxjkp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.661062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.661066    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.661338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.661082    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.661338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:20.661096    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fcc9b0-cf30-4e1a-99b7-b289f329e832-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:21.428552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:21.428521    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"
Nov 02 13:34:21.429108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:21.428554    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl" event={"ID":"b0fcc9b0-cf30-4e1a-99b7-b289f329e832","Type":"ContainerDied","Data":"694b3d0a4eafa62e4a77261f242e88577674bd87c4a8b032c49602090285e466"}
Nov 02 13:34:21.429108 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:21.428627    4928 scope.go:122] "RemoveContainer" containerID="ea4d12d3d8e41b83915c421711f2ec30e62a6e9507deae0b568c0f7001e38b5c"
Nov 02 13:34:21.484549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:21.484518    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 13:34:21.497778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:21.497750    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-61e341-7df84d5d6b-rcffl"]
Nov 02 13:34:22.497079 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:22.497040    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" path="/var/lib/kubelet/pods/b0fcc9b0-cf30-4e1a-99b7-b289f329e832/volumes"
Nov 02 13:34:23.413657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.413621    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 13:34:23.446531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.446489    4928 generic.go:356] "Generic (PLEG): container finished" podID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerID="3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814" exitCode=0
Nov 02 13:34:23.446739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.446550    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"
Nov 02 13:34:23.446739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.446570    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" event={"ID":"2da0be6d-3387-4118-8599-25e9a8026a4a","Type":"ContainerDied","Data":"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"}
Nov 02 13:34:23.446739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.446666    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w" event={"ID":"2da0be6d-3387-4118-8599-25e9a8026a4a","Type":"ContainerDied","Data":"fddbc5d58ac645d5a97a8508f12de468a243cb9c51be784d1b8d95de20c33582"}
Nov 02 13:34:23.446739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.446695    4928 scope.go:122] "RemoveContainer" containerID="3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"
Nov 02 13:34:23.456424 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.456398    4928 scope.go:122] "RemoveContainer" containerID="3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"
Nov 02 13:34:23.456704 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:23.456683    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814\": container with ID starting with 3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814 not found: ID does not exist" containerID="3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"
Nov 02 13:34:23.456779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.456713    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814"} err="failed to get container status \"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814\": rpc error: code = NotFound desc = could not find container \"3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814\": container with ID starting with 3624651ec7b72489c6446b15d00166b03aa98287117f8fbbf2cb46bfef5bb814 not found: ID does not exist"
Nov 02 13:34:23.505102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.505069    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\") pod \"2da0be6d-3387-4118-8599-25e9a8026a4a\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") "
Nov 02 13:34:23.505611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.505171    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\" (UniqueName: \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\") pod \"2da0be6d-3387-4118-8599-25e9a8026a4a\" (UID: \"2da0be6d-3387-4118-8599-25e9a8026a4a\") "
Nov 02 13:34:23.505611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.505529    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume" pod "2da0be6d-3387-4118-8599-25e9a8026a4a" (UID: "2da0be6d-3387-4118-8599-25e9a8026a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:23.507281 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.507258    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q" pod "2da0be6d-3387-4118-8599-25e9a8026a4a" (UID: "2da0be6d-3387-4118-8599-25e9a8026a4a"). InnerVolumeSpecName "kube-api-access-djv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:23.607182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.606892    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da0be6d-3387-4118-8599-25e9a8026a4a-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:23.607182 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.606930    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djv8q\" (UniqueName: \"kubernetes.io/projected/2da0be6d-3387-4118-8599-25e9a8026a4a-kube-api-access-djv8q\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:23.806498 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.806463    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 13:34:23.812093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:23.812060    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed56b9-7ffc45fcd7-dhd7w"]
Nov 02 13:34:24.497118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:24.497081    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" path="/var/lib/kubelet/pods/2da0be6d-3387-4118-8599-25e9a8026a4a/volumes"
Nov 02 13:34:29.374846 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:29.367321    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 13:34:29.374846 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:29.367594    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy" containerID="cri-o://137e15d7a6ae5598355109f8acf448800cbdfb8ca04b702bf78096d0fde1f6bd" gracePeriod=30
Nov 02 13:34:29.513054 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:29.513009    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:29.518296 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:29.518192    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:30.377512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.377472    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"]
Nov 02 13:34:30.378022 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378005    4928 cpu_manager.go:414] "containerMap: removing container" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:30.378022 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378023    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378035    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378041    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378049    4928 cpu_manager.go:414] "containerMap: removing container" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378055    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378061    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378066    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378073    4928 cpu_manager.go:414] "containerMap: removing container" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378078    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378089    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378094    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378101    4928 cpu_manager.go:414] "containerMap: removing container" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378105    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing"
Nov 02 13:34:30.378110 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378113    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378118    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378126    4928 cpu_manager.go:414] "containerMap: removing container" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378130    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378135    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378140    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378150    4928 cpu_manager.go:414] "containerMap: removing container" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378154    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378161    4928 cpu_manager.go:414] "containerMap: removing container" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378165    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378172    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378176    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378183    4928 cpu_manager.go:414] "containerMap: removing container" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378187    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378193    4928 cpu_manager.go:414] "containerMap: removing container" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378197    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378203    4928 cpu_manager.go:414] "containerMap: removing container" podUID="91ed7422-66b4-4198-9b85-956064954579" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378207    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ed7422-66b4-4198-9b85-956064954579" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378213    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378218    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378224    4928 cpu_manager.go:414] "containerMap: removing container" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378229    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378235    4928 cpu_manager.go:414] "containerMap: removing container" podUID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378239    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378245    4928 cpu_manager.go:414] "containerMap: removing container" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378250    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378258    4928 cpu_manager.go:414] "containerMap: removing container" podUID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378263    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378270    4928 cpu_manager.go:414] "containerMap: removing container" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378274    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378279    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378283    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378290    4928 cpu_manager.go:414] "containerMap: removing container" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378295    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378303    4928 cpu_manager.go:414] "containerMap: removing container" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378307    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378315    4928 cpu_manager.go:414] "containerMap: removing container" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378319    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378325    4928 cpu_manager.go:414] "containerMap: removing container" podUID="067983aa-fbee-4750-83ae-3bbbda43fc0a" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378329    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067983aa-fbee-4750-83ae-3bbbda43fc0a" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378336    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378340    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378349    4928 cpu_manager.go:414] "containerMap: removing container" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378353    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378363    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378367    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378373    4928 cpu_manager.go:414] "containerMap: removing container" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378377    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378384    4928 cpu_manager.go:414] "containerMap: removing container" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378388    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378394    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378399    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378407    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378412    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378419    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378424    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378431    4928 cpu_manager.go:414] "containerMap: removing container" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378436    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerName="maas-api"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378442    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378447    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378517    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da0be6d-3387-4118-8599-25e9a8026a4a" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378526    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378533    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378538    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378543    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a385f5-ac60-4ee4-9c30-4356b4a9f913" containerName="payload-pre-processing"
Nov 02 13:34:30.378515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378549    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378554    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9206a4db-80cd-45f5-b047-b130365277eb" containerName="main"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378560    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="6504ebd1-aaa7-4bdc-8105-6b8991468c64" containerName="payload-processing"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378566    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378571    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ed7422-66b4-4198-9b85-956064954579" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378576    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0cedb3-905a-4ba2-ad3c-b751ef431ec1" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378582    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378587    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="39df5dfc-af28-4dcb-be2c-0e01a12c9519" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378592    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7b5ac6-66a2-476c-ac73-3a200b3f0a7c" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378598    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcaa75b-7486-498d-80eb-97324ac2bdc2" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378603    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74e4841-d2ba-4a0d-82ab-a047e71ac45d" containerName="payload-processing"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378609    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378614    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378619    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378624    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ebad1b-0ba4-4ec1-959e-8e1e9f1b9a25" containerName="payload-pre-processing"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378629    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378634    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41d1b96-094b-4bc1-ad11-684ca5244607" containerName="main"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378638    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="067983aa-fbee-4750-83ae-3bbbda43fc0a" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378642    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ea8f8b-5b83-4c32-8d1c-2c04b4b85fa5" containerName="main"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378647    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378652    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6357a7d-7a1a-4a03-a008-e0e97b6413d1" containerName="main"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378657    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="10322f0b-aba5-425a-a3a8-624622fa3689" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378663    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a9f87a-1272-4591-bd5d-df59619b1f28" containerName="payload-processing"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378668    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fcc9b0-cf30-4e1a-99b7-b289f329e832" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378673    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378678    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdc46b5-2b8e-467f-b888-b269c3bbba6f" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378683    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e00ed-5726-4d6e-807e-8ac7cec0fc06" containerName="main"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378688    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a6e6e4-6d96-4f01-9b2b-37a0952cc3a7" containerName="maas-api"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378693    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378698    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378703    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378709    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ba970-d3cd-49f1-9bad-d5e345f430d5" containerName="cleanup"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378714    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbee72f1-e429-4ca4-bbf5-076da013775f" containerName="istio-proxy"
Nov 02 13:34:30.380559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.378719    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:30.383329 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.383311    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.399665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.399517    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:30.402854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.402816    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"]
Nov 02 13:34:30.412535 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.412511    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:34:30.500061 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.500016    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxrt\" (UniqueName: \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.500254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.500186    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.500254 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.500217    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.602038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.601989    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxrt\" (UniqueName: \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.602263 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.602140    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.602263 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.602174    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.602794 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.602768    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.604742 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.604717    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.628178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.628110    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxrt\" (UniqueName: \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\") pod \"maas-api-revoke-sub-0d1560a6-qds4q\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:30.696941 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:30.696906    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:31.005071 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:31.005041    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"]
Nov 02 13:34:31.007189 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:34:31.007156    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579342b9_0a69_4d1e_bdef_31becacf4f0f.slice/crio-160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c WatchSource:0}: Error finding container 160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c: Status 404 returned error can't find the container with id 160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c
Nov 02 13:34:31.536863 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:31.536800    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q" event={"ID":"579342b9-0a69-4d1e-bdef-31becacf4f0f","Type":"ContainerStarted","Data":"bdaff4e380b17593f88bff6eb4f5da32e9e1ef4e9a23c792c8bc8d791d44e783"}
Nov 02 13:34:31.537311 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:31.536886    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q" event={"ID":"579342b9-0a69-4d1e-bdef-31becacf4f0f","Type":"ContainerStarted","Data":"160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c"}
Nov 02 13:34:31.579925 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:31.579860    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q" podStartSLOduration=1.5798064429999998 podStartE2EDuration="1.579806443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:31.57827831 +0000 UTC m=+4069.227332268" watchObservedRunningTime="2026-11-02 13:34:31.579806443 +0000 UTC m=+4069.228860403"
Nov 02 13:34:32.545307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:32.545269    4928 generic.go:356] "Generic (PLEG): container finished" podID="579342b9-0a69-4d1e-bdef-31becacf4f0f" containerID="bdaff4e380b17593f88bff6eb4f5da32e9e1ef4e9a23c792c8bc8d791d44e783" exitCode=0
Nov 02 13:34:32.545734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:32.545356    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q" event={"ID":"579342b9-0a69-4d1e-bdef-31becacf4f0f","Type":"ContainerDied","Data":"bdaff4e380b17593f88bff6eb4f5da32e9e1ef4e9a23c792c8bc8d791d44e783"}
Nov 02 13:34:33.719975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.719947    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:33.837682 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.837591    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\" (UniqueName: \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\") pod \"579342b9-0a69-4d1e-bdef-31becacf4f0f\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") "
Nov 02 13:34:33.837973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.837693    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\") pod \"579342b9-0a69-4d1e-bdef-31becacf4f0f\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") "
Nov 02 13:34:33.837973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.837751    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\") pod \"579342b9-0a69-4d1e-bdef-31becacf4f0f\" (UID: \"579342b9-0a69-4d1e-bdef-31becacf4f0f\") "
Nov 02 13:34:33.838130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.838095    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca" pod "579342b9-0a69-4d1e-bdef-31becacf4f0f" (UID: "579342b9-0a69-4d1e-bdef-31becacf4f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:33.839898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.839866    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca" pod "579342b9-0a69-4d1e-bdef-31becacf4f0f" (UID: "579342b9-0a69-4d1e-bdef-31becacf4f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:33.840040 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.840015    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt" pod "579342b9-0a69-4d1e-bdef-31becacf4f0f" (UID: "579342b9-0a69-4d1e-bdef-31becacf4f0f"). InnerVolumeSpecName "kube-api-access-qmxrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:33.938868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.938838    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:33.938868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.938862    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxrt\" (UniqueName: \"kubernetes.io/projected/579342b9-0a69-4d1e-bdef-31becacf4f0f-kube-api-access-qmxrt\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:33.938868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:33.938872    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579342b9-0a69-4d1e-bdef-31becacf4f0f-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.562148 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.562111    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 13:34:34.564579 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.564546    4928 generic.go:356] "Generic (PLEG): container finished" podID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerID="30b7132eb185b3984a96b355d3131c14bb8f9e6a8442893f6909f99c247ef07d" exitCode=0
Nov 02 13:34:34.564774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.564647    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" event={"ID":"12ceb4ec-4e33-42be-81d6-e307fdb70ee9","Type":"ContainerDied","Data":"30b7132eb185b3984a96b355d3131c14bb8f9e6a8442893f6909f99c247ef07d"}
Nov 02 13:34:34.566141 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.566115    4928 generic.go:356] "Generic (PLEG): container finished" podID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerID="1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee" exitCode=0
Nov 02 13:34:34.566271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.566173    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"
Nov 02 13:34:34.566271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.566246    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" event={"ID":"8a5186cc-caf9-462b-9022-adf788b4ecec","Type":"ContainerDied","Data":"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"}
Nov 02 13:34:34.566396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.566282    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24" event={"ID":"8a5186cc-caf9-462b-9022-adf788b4ecec","Type":"ContainerDied","Data":"0aea8766aed93407182e3245bf4998686efd202c0c577b9864731149bdfeaafa"}
Nov 02 13:34:34.566396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.566305    4928 scope.go:122] "RemoveContainer" containerID="1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"
Nov 02 13:34:34.568814 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.568785    4928 generic.go:356] "Generic (PLEG): container finished" podID="29253340-c10e-49c2-9b68-d63a92f59d84" containerID="137e15d7a6ae5598355109f8acf448800cbdfb8ca04b702bf78096d0fde1f6bd" exitCode=0
Nov 02 13:34:34.568956 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.568938    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerDied","Data":"137e15d7a6ae5598355109f8acf448800cbdfb8ca04b702bf78096d0fde1f6bd"}
Nov 02 13:34:34.570239 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.570221    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q" event={"ID":"579342b9-0a69-4d1e-bdef-31becacf4f0f","Type":"ContainerDied","Data":"160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c"}
Nov 02 13:34:34.570295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.570257    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="160e199f39c0e2296940906ae3396c95d3becfb807af24ca2e30c74296d2666c"
Nov 02 13:34:34.570332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.570303    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"
Nov 02 13:34:34.578435 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.578417    4928 scope.go:122] "RemoveContainer" containerID="1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"
Nov 02 13:34:34.578719 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:34.578698    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee\": container with ID starting with 1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee not found: ID does not exist" containerID="1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"
Nov 02 13:34:34.578770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.578732    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee"} err="failed to get container status \"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee\": rpc error: code = NotFound desc = could not find container \"1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee\": container with ID starting with 1a01c484e0529af8b555f023eb51cf52eda11ae26f565ac357d56ea5243d3dee not found: ID does not exist"
Nov 02 13:34:34.578770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.578750    4928 scope.go:122] "RemoveContainer" containerID="d6d305585add06a6128fec036d7d83a5c608a4e4290657b465f1bc0a174ee1fa"
Nov 02 13:34:34.666289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.666261    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 13:34:34.688653 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.688625    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 13:34:34.749232 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.749196    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\" (UniqueName: \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\") pod \"8a5186cc-caf9-462b-9022-adf788b4ecec\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") "
Nov 02 13:34:34.749616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.749307    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\") pod \"8a5186cc-caf9-462b-9022-adf788b4ecec\" (UID: \"8a5186cc-caf9-462b-9022-adf788b4ecec\") "
Nov 02 13:34:34.749697 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.749667    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume" pod "8a5186cc-caf9-462b-9022-adf788b4ecec" (UID: "8a5186cc-caf9-462b-9022-adf788b4ecec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:34.749796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.749776    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5186cc-caf9-462b-9022-adf788b4ecec-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.751533 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.751507    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll" pod "8a5186cc-caf9-462b-9022-adf788b4ecec" (UID: "8a5186cc-caf9-462b-9022-adf788b4ecec"). InnerVolumeSpecName "kube-api-access-ds5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:34.823029 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.822944    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 13:34:34.850845 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.850792    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\") pod \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") "
Nov 02 13:34:34.851048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.850903    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.850984    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851073    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\") pod \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\" (UID: \"12ceb4ec-4e33-42be-81d6-e307fdb70ee9\") "
Nov 02 13:34:34.851189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851112    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851189 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851158    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851190    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851219    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851292    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851354 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851330    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851369    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\") pod \"29253340-c10e-49c2-9b68-d63a92f59d84\" (UID: \"29253340-c10e-49c2-9b68-d63a92f59d84\") "
Nov 02 13:34:34.851568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851403    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:34.851568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851440    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:34.851568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851518    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:34.851568 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851556    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:34.851841 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851636    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume" pod "12ceb4ec-4e33-42be-81d6-e307fdb70ee9" (UID: "12ceb4ec-4e33-42be-81d6-e307fdb70ee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:34.851925 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851908    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5ll\" (UniqueName: \"kubernetes.io/projected/8a5186cc-caf9-462b-9022-adf788b4ecec-kube-api-access-ds5ll\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.851988 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851935    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29253340-c10e-49c2-9b68-d63a92f59d84-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.851988 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851950    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.851988 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851965    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.851988 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851979    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.852218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.851993    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.852278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.852235    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:34.854282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.854256    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:34.854282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.854273    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "kube-api-access-xmzxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:34.854492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.854287    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s" pod "12ceb4ec-4e33-42be-81d6-e307fdb70ee9" (UID: "12ceb4ec-4e33-42be-81d6-e307fdb70ee9"). InnerVolumeSpecName "kube-api-access-sk64s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:34.854492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.854307    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:34.854492 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.854425    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy" pod "29253340-c10e-49c2-9b68-d63a92f59d84" (UID: "29253340-c10e-49c2-9b68-d63a92f59d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:34.917600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.917553    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 13:34:34.922899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.922863    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-61e341-54bbc99d57-kls24"]
Nov 02 13:34:34.952880 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952822    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.952880 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952871    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29253340-c10e-49c2-9b68-d63a92f59d84-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.953122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952886    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmzxv\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-kube-api-access-xmzxv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.953122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952904    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29253340-c10e-49c2-9b68-d63a92f59d84-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.953122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952914    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/12ceb4ec-4e33-42be-81d6-e307fdb70ee9-kube-api-access-sk64s\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:34.953122 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:34.952923    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29253340-c10e-49c2-9b68-d63a92f59d84-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:35.578914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.578878    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"
Nov 02 13:34:35.579193 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.579162    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx" event={"ID":"12ceb4ec-4e33-42be-81d6-e307fdb70ee9","Type":"ContainerDied","Data":"395386ca730610849b2ad252b7d706b210f2b80f71963ce8b8c0a8d52ec304e1"}
Nov 02 13:34:35.579266 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.579237    4928 scope.go:122] "RemoveContainer" containerID="30b7132eb185b3984a96b355d3131c14bb8f9e6a8442893f6909f99c247ef07d"
Nov 02 13:34:35.581973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.581947    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"
Nov 02 13:34:35.582113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.582088    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh" event={"ID":"29253340-c10e-49c2-9b68-d63a92f59d84","Type":"ContainerDied","Data":"dbd70fb0144e96db4751485170d78becbbc1511af21ad9a285c355aca9ae5d1c"}
Nov 02 13:34:35.589645 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.589621    4928 scope.go:122] "RemoveContainer" containerID="137e15d7a6ae5598355109f8acf448800cbdfb8ca04b702bf78096d0fde1f6bd"
Nov 02 13:34:35.636416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.636376    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 13:34:35.648799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.648764    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-62grh"]
Nov 02 13:34:35.676407 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.676371    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 13:34:35.682048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:35.682020    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-61e341-55d84b94dc-78ljx"]
Nov 02 13:34:36.497985 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:36.497947    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" path="/var/lib/kubelet/pods/12ceb4ec-4e33-42be-81d6-e307fdb70ee9/volumes"
Nov 02 13:34:36.498443 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:36.498427    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" path="/var/lib/kubelet/pods/29253340-c10e-49c2-9b68-d63a92f59d84/volumes"
Nov 02 13:34:36.498804 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:36.498791    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" path="/var/lib/kubelet/pods/8a5186cc-caf9-462b-9022-adf788b4ecec/volumes"
Nov 02 13:34:39.729408 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:39.729367    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:34:41.845328 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.845283    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"]
Nov 02 13:34:41.846062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846039    4928 cpu_manager.go:414] "containerMap: removing container" podUID="579342b9-0a69-4d1e-bdef-31becacf4f0f" containerName="revoke-keys"
Nov 02 13:34:41.846130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846066    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579342b9-0a69-4d1e-bdef-31becacf4f0f" containerName="revoke-keys"
Nov 02 13:34:41.846130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846095    4928 cpu_manager.go:414] "containerMap: removing container" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:41.846130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846104    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474cc566-a72a-4bf1-b668-eb05ae7985a8" containerName="istio-proxy"
Nov 02 13:34:41.846130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846119    4928 cpu_manager.go:414] "containerMap: removing container" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:34:41.846130 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846129    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846144    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846153    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe9fac3-d823-4655-a6eb-27f6235d35e8" containerName="cleanup"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846177    4928 cpu_manager.go:414] "containerMap: removing container" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846186    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846199    4928 cpu_manager.go:414] "containerMap: removing container" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846207    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846221    4928 cpu_manager.go:414] "containerMap: removing container" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846230    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d55fcd-fa53-452a-bb7e-1134b9581b6a" containerName="istio-proxy"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846262    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing"
Nov 02 13:34:41.846298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846271    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846398    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5186cc-caf9-462b-9022-adf788b4ecec" containerName="payload-processing"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846412    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca0717d-8b99-4ee1-86bd-25fa117c9d1d" containerName="cleanup"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846426    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846438    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bca6ff-5b72-4cc8-be02-badc173e9845" containerName="cleanup"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846449    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="579342b9-0a69-4d1e-bdef-31becacf4f0f" containerName="revoke-keys"
Nov 02 13:34:41.846603 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.846466    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ceb4ec-4e33-42be-81d6-e307fdb70ee9" containerName="payload-pre-processing"
Nov 02 13:34:41.851769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.851743    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:41.877136 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.877099    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:41.877283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.877159    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:34:41.895530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.895491    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"]
Nov 02 13:34:41.950146 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.950111    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:41.950324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.950155    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8h6\" (UniqueName: \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:41.950324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:41.950266    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.052301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.052252    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.052502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.052332    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8h6\" (UniqueName: \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.052502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.052389    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.053070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.053043    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.054755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.054735    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.076611 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.076582    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8h6\" (UniqueName: \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\") pod \"maas-api-revoke-sub-36e70ab4-vdsm8\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.170666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.170563    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:42.540951 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.538258    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"]
Nov 02 13:34:42.561533 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:34:42.561502    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f395794_bb0b_4e24_a343_71a2d56b7bfd.slice/crio-0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff WatchSource:0}: Error finding container 0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff: Status 404 returned error can't find the container with id 0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff
Nov 02 13:34:42.658846 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:42.658755    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8" event={"ID":"3f395794-bb0b-4e24-a343-71a2d56b7bfd","Type":"ContainerStarted","Data":"0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff"}
Nov 02 13:34:43.461898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:43.461853    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 13:34:43.669490 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:43.669457    4928 generic.go:356] "Generic (PLEG): container finished" podID="3f395794-bb0b-4e24-a343-71a2d56b7bfd" containerID="996b74bbbb4d570f217f4c02fe7ed57e0dae1561e7e06b32511d282d75e27827" exitCode=0
Nov 02 13:34:43.669677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:43.669540    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8" event={"ID":"3f395794-bb0b-4e24-a343-71a2d56b7bfd","Type":"ContainerDied","Data":"996b74bbbb4d570f217f4c02fe7ed57e0dae1561e7e06b32511d282d75e27827"}
Nov 02 13:34:44.823939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.823909    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:44.889569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.889535    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\") pod \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") "
Nov 02 13:34:44.889773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.889616    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\" (UniqueName: \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\") pod \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") "
Nov 02 13:34:44.889773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.889713    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\") pod \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\" (UID: \"3f395794-bb0b-4e24-a343-71a2d56b7bfd\") "
Nov 02 13:34:44.890058 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.890036    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca" pod "3f395794-bb0b-4e24-a343-71a2d56b7bfd" (UID: "3f395794-bb0b-4e24-a343-71a2d56b7bfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:44.890157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.890137    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:44.891856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.891812    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6" pod "3f395794-bb0b-4e24-a343-71a2d56b7bfd" (UID: "3f395794-bb0b-4e24-a343-71a2d56b7bfd"). InnerVolumeSpecName "kube-api-access-vd8h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:44.891856 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.891847    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca" pod "3f395794-bb0b-4e24-a343-71a2d56b7bfd" (UID: "3f395794-bb0b-4e24-a343-71a2d56b7bfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:44.991223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.991196    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f395794-bb0b-4e24-a343-71a2d56b7bfd-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:44.991223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:44.991224    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8h6\" (UniqueName: \"kubernetes.io/projected/3f395794-bb0b-4e24-a343-71a2d56b7bfd-kube-api-access-vd8h6\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:45.685848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:45.685751    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"
Nov 02 13:34:45.685848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:45.685785    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8" event={"ID":"3f395794-bb0b-4e24-a343-71a2d56b7bfd","Type":"ContainerDied","Data":"0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff"}
Nov 02 13:34:45.685848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:45.685846    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0143e6665da1cceff8dc745a6b2bfee3bad16eb39ec86a9003d3dda2e14905ff"
Nov 02 13:34:48.462261 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:48.462193    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing" containerID="cri-o://1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a" gracePeriod=25
Nov 02 13:34:49.141632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:49.141596    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:49.844187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:49.844137    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main" containerID="cri-o://0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323" gracePeriod=45
Nov 02 13:34:50.110035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.109965    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 13:34:50.250989 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.250955    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251188 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251005    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251188 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251137    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\" (UniqueName: \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251188 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251165    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251226    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251258    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\") pod \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\" (UID: \"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e\") "
Nov 02 13:34:50.251393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251272    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.251596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251543    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.251670 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251612    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.251809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251788    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.251914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251814    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.251914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.251854    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.253447 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.253418    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "kube-api-access-b5hkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:50.253563 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.253441    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:50.253563 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.253504    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm" pod "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" (UID: "d4312071-44b5-4b2c-9e6c-5b9d8c677f3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.353559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.353517    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hkr\" (UniqueName: \"kubernetes.io/projected/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-kube-api-access-b5hkr\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.353559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.353558    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.353784 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.353572    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.721194 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.721157    4928 generic.go:356] "Generic (PLEG): container finished" podID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerID="0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323" exitCode=0
Nov 02 13:34:50.721393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.721214    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"
Nov 02 13:34:50.721393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.721238    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" event={"ID":"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e","Type":"ContainerDied","Data":"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"}
Nov 02 13:34:50.721393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.721282    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k" event={"ID":"d4312071-44b5-4b2c-9e6c-5b9d8c677f3e","Type":"ContainerDied","Data":"1e62a1201afcfda337ee95e1685a738a68ad2c85513b73230c69999c39c0d188"}
Nov 02 13:34:50.721393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.721301    4928 scope.go:122] "RemoveContainer" containerID="0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"
Nov 02 13:34:50.733394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.733361    4928 scope.go:122] "RemoveContainer" containerID="0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"
Nov 02 13:34:50.733709 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:34:50.733681    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323\": container with ID starting with 0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323 not found: ID does not exist" containerID="0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"
Nov 02 13:34:50.733854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.733725    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323"} err="failed to get container status \"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323\": rpc error: code = NotFound desc = could not find container \"0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323\": container with ID starting with 0ddd060061d528e53d32a66740d8edc9bdd14571e6ecef231539e4cb1806f323 not found: ID does not exist"
Nov 02 13:34:50.776967 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.776894    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 13:34:50.790197 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:50.790169    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-2-w2-95ee26-kserve-5695b88fcb-kzz5k"]
Nov 02 13:34:52.497180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:52.497142    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" path="/var/lib/kubelet/pods/d4312071-44b5-4b2c-9e6c-5b9d8c677f3e/volumes"
Nov 02 13:34:59.141888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:34:59.141769    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:35:08.737525 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.737489    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 13:35:08.765632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.765601    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\") pod \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") "
Nov 02 13:35:08.765786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.765702    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\" (UniqueName: \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\") pod \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\" (UID: \"57a98d9e-e77a-436c-923d-3f8a9ae05b15\") "
Nov 02 13:35:08.765998 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.765973    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume" pod "57a98d9e-e77a-436c-923d-3f8a9ae05b15" (UID: "57a98d9e-e77a-436c-923d-3f8a9ae05b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:08.766137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.766120    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a98d9e-e77a-436c-923d-3f8a9ae05b15-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:08.767925 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.767895    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf" pod "57a98d9e-e77a-436c-923d-3f8a9ae05b15" (UID: "57a98d9e-e77a-436c-923d-3f8a9ae05b15"). InnerVolumeSpecName "kube-api-access-pg9cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:08.838766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.838729    4928 generic.go:356] "Generic (PLEG): container finished" podID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerID="1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a" exitCode=0
Nov 02 13:35:08.839010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.838782    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"
Nov 02 13:35:08.839010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.838849    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" event={"ID":"57a98d9e-e77a-436c-923d-3f8a9ae05b15","Type":"ContainerDied","Data":"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"}
Nov 02 13:35:08.839010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.838909    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw" event={"ID":"57a98d9e-e77a-436c-923d-3f8a9ae05b15","Type":"ContainerDied","Data":"2f769271270670f69aa53754504bb38205220fbb0f6bccef518ed52d7a2c0b30"}
Nov 02 13:35:08.839010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.838934    4928 scope.go:122] "RemoveContainer" containerID="1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"
Nov 02 13:35:08.851087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.851057    4928 scope.go:122] "RemoveContainer" containerID="1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"
Nov 02 13:35:08.851415 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:35:08.851391    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a\": container with ID starting with 1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a not found: ID does not exist" containerID="1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"
Nov 02 13:35:08.851482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.851424    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a"} err="failed to get container status \"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a\": rpc error: code = NotFound desc = could not find container \"1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a\": container with ID starting with 1f95813a6ecc4699a8c8cee50d2923baea63cf150e533fd4e52009b6b832d49a not found: ID does not exist"
Nov 02 13:35:08.867195 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.867163    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9cf\" (UniqueName: \"kubernetes.io/projected/57a98d9e-e77a-436c-923d-3f8a9ae05b15-kube-api-access-pg9cf\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:08.959008 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.958961    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 13:35:08.974299 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:08.974202    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-95ee26-677585b5b7-q7wrw"]
Nov 02 13:35:09.729115 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:35:09.729075    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:35:10.496456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:10.496419    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" path="/var/lib/kubelet/pods/57a98d9e-e77a-436c-923d-3f8a9ae05b15/volumes"
Nov 02 13:35:12.652781 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.652735    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"]
Nov 02 13:35:12.653238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653221    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3f395794-bb0b-4e24-a343-71a2d56b7bfd" containerName="revoke-keys"
Nov 02 13:35:12.653238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653235    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f395794-bb0b-4e24-a343-71a2d56b7bfd" containerName="revoke-keys"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653247    4928 cpu_manager.go:414] "containerMap: removing container" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653253    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653271    4928 cpu_manager.go:414] "containerMap: removing container" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653277    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653295    4928 cpu_manager.go:414] "containerMap: removing container" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:35:12.653323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653302    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:35:12.653514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653374    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f395794-bb0b-4e24-a343-71a2d56b7bfd" containerName="revoke-keys"
Nov 02 13:35:12.653514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653382    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4312071-44b5-4b2c-9e6c-5b9d8c677f3e" containerName="main"
Nov 02 13:35:12.653514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653391    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="29253340-c10e-49c2-9b68-d63a92f59d84" containerName="istio-proxy"
Nov 02 13:35:12.653514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.653398    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a98d9e-e77a-436c-923d-3f8a9ae05b15" containerName="payload-pre-processing"
Nov 02 13:35:12.658113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.658089    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.674736 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.674707    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:35:12.688111 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.688086    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:35:12.692545 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.692518    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"]
Nov 02 13:35:12.711335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.711295    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.711502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.711399    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.711502 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.711437    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nh24\" (UniqueName: \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.812895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.812850    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.812895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.812905    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.813195 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.813022    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nh24\" (UniqueName: \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.813698 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.813674    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.815361 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.815330    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.838416 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.838382    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nh24\" (UniqueName: \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\") pod \"maas-api-revoke-sub-ea1ea3d6-95wtb\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:12.971516 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:12.971432    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:13.302801 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:13.302773    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"]
Nov 02 13:35:13.304992 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:35:13.304964    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345feb12_e066_4a6b_9f46_5006d8563049.slice/crio-e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf WatchSource:0}: Error finding container e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf: Status 404 returned error can't find the container with id e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf
Nov 02 13:35:13.878161 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:13.878123    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb" event={"ID":"345feb12-e066-4a6b-9f46-5006d8563049","Type":"ContainerStarted","Data":"f6a01084da4ed883cd6434256dd631bb19dfb4a0742ec0ed04b3f20216561a4c"}
Nov 02 13:35:13.878541 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:13.878185    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb" event={"ID":"345feb12-e066-4a6b-9f46-5006d8563049","Type":"ContainerStarted","Data":"e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf"}
Nov 02 13:35:13.920367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:13.920265    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb" podStartSLOduration=1.920251715 podStartE2EDuration="1.920251715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:35:13.920196762 +0000 UTC m=+4111.569250722" watchObservedRunningTime="2026-11-02 13:35:13.920251715 +0000 UTC m=+4111.569305673"
Nov 02 13:35:14.885641 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:14.885605    4928 generic.go:356] "Generic (PLEG): container finished" podID="345feb12-e066-4a6b-9f46-5006d8563049" containerID="f6a01084da4ed883cd6434256dd631bb19dfb4a0742ec0ed04b3f20216561a4c" exitCode=0
Nov 02 13:35:14.886144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:14.885687    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb" event={"ID":"345feb12-e066-4a6b-9f46-5006d8563049","Type":"ContainerDied","Data":"f6a01084da4ed883cd6434256dd631bb19dfb4a0742ec0ed04b3f20216561a4c"}
Nov 02 13:35:16.098966 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.098937    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.151749    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.151991    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy" containerID="cri-o://ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0" gracePeriod=30
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.152564    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\") pod \"345feb12-e066-4a6b-9f46-5006d8563049\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") "
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.152686    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\" (UniqueName: \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\") pod \"345feb12-e066-4a6b-9f46-5006d8563049\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") "
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.152780    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\") pod \"345feb12-e066-4a6b-9f46-5006d8563049\" (UID: \"345feb12-e066-4a6b-9f46-5006d8563049\") "
Nov 02 13:35:16.171211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.153564    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca" pod "345feb12-e066-4a6b-9f46-5006d8563049" (UID: "345feb12-e066-4a6b-9f46-5006d8563049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:16.176694 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.175030    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca" pod "345feb12-e066-4a6b-9f46-5006d8563049" (UID: "345feb12-e066-4a6b-9f46-5006d8563049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:16.181446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.179532    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24" pod "345feb12-e066-4a6b-9f46-5006d8563049" (UID: "345feb12-e066-4a6b-9f46-5006d8563049"). InnerVolumeSpecName "kube-api-access-6nh24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:16.288674 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.288617    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:16.291095 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.291051    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nh24\" (UniqueName: \"kubernetes.io/projected/345feb12-e066-4a6b-9f46-5006d8563049-kube-api-access-6nh24\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:16.292373 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.292351    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/345feb12-e066-4a6b-9f46-5006d8563049-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:16.904271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.904187    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb" event={"ID":"345feb12-e066-4a6b-9f46-5006d8563049","Type":"ContainerDied","Data":"e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf"}
Nov 02 13:35:16.904271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.904229    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"
Nov 02 13:35:16.904271 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:16.904243    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e434ca0147102b3aa8b156e7efc68de461bd60f71b649596012832e817561dcf"
Nov 02 13:35:21.430759 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.430701    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 13:35:21.451951 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.451923    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.451957    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.451997    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452075 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452052    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452089    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452139    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452181    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452193    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:21.452224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452219    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452331    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\") pod \"88845a7d-45bb-42e0-ba81-1c17f3791c95\" (UID: \"88845a7d-45bb-42e0-ba81-1c17f3791c95\") "
Nov 02 13:35:21.452505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452346    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:21.452505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452464    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:21.452681 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452527    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:21.452736 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452596    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:21.452854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452815    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.452854 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452857    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88845a7d-45bb-42e0-ba81-1c17f3791c95-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.453072 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452872    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.453072 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452888    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.453072 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.452903    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.454657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.454629    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "kube-api-access-lnt4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:21.454754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.454654    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:21.454754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.454729    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:21.454872 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.454803    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo" pod "88845a7d-45bb-42e0-ba81-1c17f3791c95" (UID: "88845a7d-45bb-42e0-ba81-1c17f3791c95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:35:21.553668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.553626    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnt4q\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-kube-api-access-lnt4q\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.553668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.553661    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.553668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.553674    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.553944 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.553687    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88845a7d-45bb-42e0-ba81-1c17f3791c95-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:21.936631 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.936593    4928 generic.go:356] "Generic (PLEG): container finished" podID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerID="ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0" exitCode=0
Nov 02 13:35:21.936815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.936641    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"
Nov 02 13:35:21.936815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.936675    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" event={"ID":"88845a7d-45bb-42e0-ba81-1c17f3791c95","Type":"ContainerDied","Data":"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"}
Nov 02 13:35:21.936815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.936743    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm" event={"ID":"88845a7d-45bb-42e0-ba81-1c17f3791c95","Type":"ContainerDied","Data":"13a077d70267dddb1a2f72ec4bc3fc9af84130b60a8976df7ba94a4248244cf9"}
Nov 02 13:35:21.936815 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.936768    4928 scope.go:122] "RemoveContainer" containerID="ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"
Nov 02 13:35:21.947324 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.947302    4928 scope.go:122] "RemoveContainer" containerID="ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"
Nov 02 13:35:21.947644 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:35:21.947620    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0\": container with ID starting with ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0 not found: ID does not exist" containerID="ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"
Nov 02 13:35:21.947693 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.947655    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0"} err="failed to get container status \"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0\": rpc error: code = NotFound desc = could not find container \"ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0\": container with ID starting with ce2c75b96db8bab482ca19567d7888ffbde5b4ee9c8084877fa6e63a5f924cb0 not found: ID does not exist"
Nov 02 13:35:21.992990 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:21.992959    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 13:35:22.006451 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:22.006414    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-v8nzm"]
Nov 02 13:35:22.497187 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:22.497147    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" path="/var/lib/kubelet/pods/88845a7d-45bb-42e0-ba81-1c17f3791c95/volumes"
Nov 02 13:35:23.430201 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430162    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"]
Nov 02 13:35:23.430677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430662    4928 cpu_manager.go:414] "containerMap: removing container" podUID="345feb12-e066-4a6b-9f46-5006d8563049" containerName="revoke-keys"
Nov 02 13:35:23.430732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430679    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345feb12-e066-4a6b-9f46-5006d8563049" containerName="revoke-keys"
Nov 02 13:35:23.430732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430701    4928 cpu_manager.go:414] "containerMap: removing container" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy"
Nov 02 13:35:23.430732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430708    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy"
Nov 02 13:35:23.430847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430788    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="88845a7d-45bb-42e0-ba81-1c17f3791c95" containerName="istio-proxy"
Nov 02 13:35:23.430847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.430808    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="345feb12-e066-4a6b-9f46-5006d8563049" containerName="revoke-keys"
Nov 02 13:35:23.435344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.435318    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.455715 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.455677    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"]
Nov 02 13:35:23.465644 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.465614    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:35:23.465890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.465851    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:35:23.473277 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.473250    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.473499 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.473473    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.473548 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.473529    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sj7m\" (UniqueName: \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.574811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.574767    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.574811 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.574816    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sj7m\" (UniqueName: \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.575485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.574870    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.575666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.575643    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.577398 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.577372    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.599377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.599349    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sj7m\" (UniqueName: \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\") pod \"maas-api-revoke-sub-1fefdb5b-2qtmm\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:23.748613 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:23.748575    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:24.051527 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:24.051494    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"]
Nov 02 13:35:24.053683 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:35:24.053653    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114647d3_4b75_40a7_922f_0e2e9b72a797.slice/crio-ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019 WatchSource:0}: Error finding container ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019: Status 404 returned error can't find the container with id ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019
Nov 02 13:35:24.963713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:24.963196    4928 generic.go:356] "Generic (PLEG): container finished" podID="114647d3-4b75-40a7-922f-0e2e9b72a797" containerID="d51e7a9e6fc0054f1fba26f4e029f756e499c64cce7aedd5db9a49950475a93c" exitCode=0
Nov 02 13:35:24.963713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:24.963300    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm" event={"ID":"114647d3-4b75-40a7-922f-0e2e9b72a797","Type":"ContainerDied","Data":"d51e7a9e6fc0054f1fba26f4e029f756e499c64cce7aedd5db9a49950475a93c"}
Nov 02 13:35:24.963713 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:24.963354    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm" event={"ID":"114647d3-4b75-40a7-922f-0e2e9b72a797","Type":"ContainerStarted","Data":"ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019"}
Nov 02 13:35:26.132319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.132288    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:26.206870 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.206803    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\") pod \"114647d3-4b75-40a7-922f-0e2e9b72a797\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") "
Nov 02 13:35:26.207076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.206876    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\") pod \"114647d3-4b75-40a7-922f-0e2e9b72a797\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") "
Nov 02 13:35:26.207076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.206942    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\" (UniqueName: \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\") pod \"114647d3-4b75-40a7-922f-0e2e9b72a797\" (UID: \"114647d3-4b75-40a7-922f-0e2e9b72a797\") "
Nov 02 13:35:26.207387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.207354    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca" pod "114647d3-4b75-40a7-922f-0e2e9b72a797" (UID: "114647d3-4b75-40a7-922f-0e2e9b72a797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:26.209597 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.209567    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca" pod "114647d3-4b75-40a7-922f-0e2e9b72a797" (UID: "114647d3-4b75-40a7-922f-0e2e9b72a797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:26.209739 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.209650    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m" pod "114647d3-4b75-40a7-922f-0e2e9b72a797" (UID: "114647d3-4b75-40a7-922f-0e2e9b72a797"). InnerVolumeSpecName "kube-api-access-7sj7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:26.309084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.309047    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:26.309351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.309332    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114647d3-4b75-40a7-922f-0e2e9b72a797-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:26.309481 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.309462    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sj7m\" (UniqueName: \"kubernetes.io/projected/114647d3-4b75-40a7-922f-0e2e9b72a797-kube-api-access-7sj7m\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:26.976262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.976229    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"
Nov 02 13:35:26.976463 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.976263    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm" event={"ID":"114647d3-4b75-40a7-922f-0e2e9b72a797","Type":"ContainerDied","Data":"ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019"}
Nov 02 13:35:26.976463 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:26.976322    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee7710f3029909d60c8710c0f7d010ad71f9dfab624e606f83bed83429e5a019"
Nov 02 13:35:33.922196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.922158    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"]
Nov 02 13:35:33.922618 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.922604    4928 cpu_manager.go:414] "containerMap: removing container" podUID="114647d3-4b75-40a7-922f-0e2e9b72a797" containerName="revoke-keys"
Nov 02 13:35:33.922683 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.922619    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114647d3-4b75-40a7-922f-0e2e9b72a797" containerName="revoke-keys"
Nov 02 13:35:33.922721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.922691    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="114647d3-4b75-40a7-922f-0e2e9b72a797" containerName="revoke-keys"
Nov 02 13:35:33.926127 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.926107    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:33.958255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.958225    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:35:33.958473 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.958279    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:35:33.962686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:33.962662    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"]
Nov 02 13:35:34.005094 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.005063    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.005256 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.005119    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnc29\" (UniqueName: \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.005374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.005345    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.106310 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.106264    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.106515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.106350    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.106515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.106399    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc29\" (UniqueName: \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.107144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.107115    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.108700 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.108681    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.130907 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.130880    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnc29\" (UniqueName: \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\") pod \"maas-api-revoke-sub-4a55b1f1-rdwm9\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.239232 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.239193    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:34.544229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:34.544198    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"]
Nov 02 13:35:34.546492 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:35:34.546458    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7ec296_0dc8_46c0_be1b_1384813b5ce3.slice/crio-fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3 WatchSource:0}: Error finding container fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3: Status 404 returned error can't find the container with id fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3
Nov 02 13:35:35.028716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:35.028676    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9" event={"ID":"2c7ec296-0dc8-46c0-be1b-1384813b5ce3","Type":"ContainerStarted","Data":"cded59bd18433e289491ebe3f51aaf4404fffcf2ef612220ad047741df4c2942"}
Nov 02 13:35:35.029137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:35.028742    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9" event={"ID":"2c7ec296-0dc8-46c0-be1b-1384813b5ce3","Type":"ContainerStarted","Data":"fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3"}
Nov 02 13:35:35.080272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:35.080208    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9" podStartSLOduration=2.08019456 podStartE2EDuration="2.08019456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:35:35.07824367 +0000 UTC m=+4132.727297630" watchObservedRunningTime="2026-11-02 13:35:35.08019456 +0000 UTC m=+4132.729248522"
Nov 02 13:35:36.037250 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:36.037210    4928 generic.go:356] "Generic (PLEG): container finished" podID="2c7ec296-0dc8-46c0-be1b-1384813b5ce3" containerID="cded59bd18433e289491ebe3f51aaf4404fffcf2ef612220ad047741df4c2942" exitCode=0
Nov 02 13:35:36.037702 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:36.037295    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9" event={"ID":"2c7ec296-0dc8-46c0-be1b-1384813b5ce3","Type":"ContainerDied","Data":"cded59bd18433e289491ebe3f51aaf4404fffcf2ef612220ad047741df4c2942"}
Nov 02 13:35:37.195133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.195104    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:37.342128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.342036    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\") pod \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") "
Nov 02 13:35:37.342128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.342093    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\" (UniqueName: \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\") pod \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") "
Nov 02 13:35:37.342382 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.342141    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\") pod \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\" (UID: \"2c7ec296-0dc8-46c0-be1b-1384813b5ce3\") "
Nov 02 13:35:37.342453 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.342419    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca" pod "2c7ec296-0dc8-46c0-be1b-1384813b5ce3" (UID: "2c7ec296-0dc8-46c0-be1b-1384813b5ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:37.342690 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.342664    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:37.344365 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.344335    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca" pod "2c7ec296-0dc8-46c0-be1b-1384813b5ce3" (UID: "2c7ec296-0dc8-46c0-be1b-1384813b5ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:37.344468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.344340    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29" pod "2c7ec296-0dc8-46c0-be1b-1384813b5ce3" (UID: "2c7ec296-0dc8-46c0-be1b-1384813b5ce3"). InnerVolumeSpecName "kube-api-access-dnc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:37.444343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.444293    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnc29\" (UniqueName: \"kubernetes.io/projected/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-kube-api-access-dnc29\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:37.444343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:37.444342    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c7ec296-0dc8-46c0-be1b-1384813b5ce3-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:38.051880 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:38.051818    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"
Nov 02 13:35:38.052091 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:38.051817    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9" event={"ID":"2c7ec296-0dc8-46c0-be1b-1384813b5ce3","Type":"ContainerDied","Data":"fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3"}
Nov 02 13:35:38.052091 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:38.051979    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe70bc01a8442f3e60fc2bf52f9fd02edfabaee9d047a94d4ace0b36cd0d49a3"
Nov 02 13:35:39.731559 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:35:39.731521    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:35:45.513668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:45.513633    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:35:45.514164 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:45.513867    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy" containerID="cri-o://e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811" gracePeriod=30
Nov 02 13:35:50.781170 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.781092    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:35:50.889666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889637    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.889851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889677    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.889851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889707    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.889851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889754    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.889851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889783    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.889851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889812    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.890157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889906    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.890157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.889946    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.890157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890018    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:50.890157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890034    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\") pod \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\" (UID: \"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33\") "
Nov 02 13:35:50.890367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890028    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:50.890367 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890249    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:50.890461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890374    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:50.890556 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890528    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:50.890649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890596    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.890649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890616    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.890649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890631    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.890649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.890646    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.892167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.892142    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:35:50.892167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.892149    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:50.892331 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.892255    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "kube-api-access-t6l8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:50.892331 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.892265    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy" pod "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" (UID: "8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:50.991888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.991817    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.991888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.991885    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.991888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.991895    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6l8m\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-kube-api-access-t6l8m\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.992133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.991905    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:50.992133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:50.991914    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:35:51.129677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.129568    4928 generic.go:356] "Generic (PLEG): container finished" podID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerID="e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811" exitCode=0
Nov 02 13:35:51.129677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.129617    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"
Nov 02 13:35:51.129955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.129663    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" event={"ID":"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33","Type":"ContainerDied","Data":"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"}
Nov 02 13:35:51.129955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.129728    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf" event={"ID":"8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33","Type":"ContainerDied","Data":"cc2e129c75fc67783cb91ae14f08d3485f027ffd9da86c754aaf31379ae28c11"}
Nov 02 13:35:51.129955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.129758    4928 scope.go:122] "RemoveContainer" containerID="e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"
Nov 02 13:35:51.140256 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.140239    4928 scope.go:122] "RemoveContainer" containerID="e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"
Nov 02 13:35:51.140522 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:35:51.140502    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811\": container with ID starting with e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811 not found: ID does not exist" containerID="e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"
Nov 02 13:35:51.140583 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.140530    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811"} err="failed to get container status \"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811\": rpc error: code = NotFound desc = could not find container \"e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811\": container with ID starting with e76528cbe5a7a3bffd5c5726501bb03be820afc8e2c1d8ea6879fbf0a885b811 not found: ID does not exist"
Nov 02 13:35:51.186211 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.186172    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:35:51.200725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:51.200690    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2t5gf"]
Nov 02 13:35:52.496500 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:35:52.496465    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" path="/var/lib/kubelet/pods/8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33/volumes"
Nov 02 13:36:09.729786 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:36:09.729742    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:36:21.092484 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.092448    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:36:21.095043 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.092681    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" podUID="2fff3afe-617b-43e3-b764-1b66db982d58" containerName="maas-api" containerID="cri-o://2dbe415836f97d9bb2778eb4eb569407f6c8de0d4333e6114e6b8a23135a2b26" gracePeriod=30
Nov 02 13:36:21.350890 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.350771    4928 generic.go:356] "Generic (PLEG): container finished" podID="2fff3afe-617b-43e3-b764-1b66db982d58" containerID="2dbe415836f97d9bb2778eb4eb569407f6c8de0d4333e6114e6b8a23135a2b26" exitCode=0
Nov 02 13:36:21.351112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.350911    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" event={"ID":"2fff3afe-617b-43e3-b764-1b66db982d58","Type":"ContainerDied","Data":"2dbe415836f97d9bb2778eb4eb569407f6c8de0d4333e6114e6b8a23135a2b26"}
Nov 02 13:36:21.376725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.376695    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:36:21.447209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447174    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\" (UniqueName: \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\") pod \"2fff3afe-617b-43e3-b764-1b66db982d58\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") "
Nov 02 13:36:21.447414 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447221    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\") pod \"2fff3afe-617b-43e3-b764-1b66db982d58\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") "
Nov 02 13:36:21.447414 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447366    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\") pod \"2fff3afe-617b-43e3-b764-1b66db982d58\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") "
Nov 02 13:36:21.447606 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447472    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\") pod \"2fff3afe-617b-43e3-b764-1b66db982d58\" (UID: \"2fff3afe-617b-43e3-b764-1b66db982d58\") "
Nov 02 13:36:21.447769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447743    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca" pod "2fff3afe-617b-43e3-b764-1b66db982d58" (UID: "2fff3afe-617b-43e3-b764-1b66db982d58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:21.447950 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.447924    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fff3afe-617b-43e3-b764-1b66db982d58-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:36:21.449467 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.449439    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w" pod "2fff3afe-617b-43e3-b764-1b66db982d58" (UID: "2fff3afe-617b-43e3-b764-1b66db982d58"). InnerVolumeSpecName "kube-api-access-q6c2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:21.449600 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.449488    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs" pod "2fff3afe-617b-43e3-b764-1b66db982d58" (UID: "2fff3afe-617b-43e3-b764-1b66db982d58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:21.450086 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.450068    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls" pod "2fff3afe-617b-43e3-b764-1b66db982d58" (UID: "2fff3afe-617b-43e3-b764-1b66db982d58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:21.549520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.549315    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6c2w\" (UniqueName: \"kubernetes.io/projected/2fff3afe-617b-43e3-b764-1b66db982d58-kube-api-access-q6c2w\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:36:21.549520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.549357    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:36:21.549520 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:21.549377    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fff3afe-617b-43e3-b764-1b66db982d58-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:36:22.357896 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.357863    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"
Nov 02 13:36:22.358449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.357862    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg" event={"ID":"2fff3afe-617b-43e3-b764-1b66db982d58","Type":"ContainerDied","Data":"a0378560a74cf4467a48069752e1ca44eb2854a66383d8a31132ae086bab6f64"}
Nov 02 13:36:22.358449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.358002    4928 scope.go:122] "RemoveContainer" containerID="2dbe415836f97d9bb2778eb4eb569407f6c8de0d4333e6114e6b8a23135a2b26"
Nov 02 13:36:22.415812 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.415775    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:36:22.428667 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.428635    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e3396f-5844455fc9-gblgg"]
Nov 02 13:36:22.496131 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:22.496095    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fff3afe-617b-43e3-b764-1b66db982d58" path="/var/lib/kubelet/pods/2fff3afe-617b-43e3-b764-1b66db982d58/volumes"
Nov 02 13:36:36.241387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:36.241352    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 13:36:39.729037 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:36:39.728987    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:36:41.241782 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:41.241734    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing" containerID="cri-o://d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a" gracePeriod=25
Nov 02 13:36:43.404409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:43.404375    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:36:43.418173 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:43.418150    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:36:49.612133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:49.612084    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:59.612718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:59.612672    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:59.783949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:59.783912    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 13:36:59.784213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:36:59.784186    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy" containerID="cri-o://6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c" gracePeriod=30
Nov 02 13:37:00.712846 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.712796    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"]
Nov 02 13:37:00.713441 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713420    4928 cpu_manager.go:414] "containerMap: removing container" podUID="2c7ec296-0dc8-46c0-be1b-1384813b5ce3" containerName="revoke-keys"
Nov 02 13:37:00.713441 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713441    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7ec296-0dc8-46c0-be1b-1384813b5ce3" containerName="revoke-keys"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713466    4928 cpu_manager.go:414] "containerMap: removing container" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713472    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713489    4928 cpu_manager.go:414] "containerMap: removing container" podUID="2fff3afe-617b-43e3-b764-1b66db982d58" containerName="maas-api"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713496    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fff3afe-617b-43e3-b764-1b66db982d58" containerName="maas-api"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713562    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3f42a5-3bcd-4e40-aa0a-ef7a67b53f33" containerName="istio-proxy"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713573    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fff3afe-617b-43e3-b764-1b66db982d58" containerName="maas-api"
Nov 02 13:37:00.713582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.713582    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7ec296-0dc8-46c0-be1b-1384813b5ce3" containerName="revoke-keys"
Nov 02 13:37:00.717023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.716998    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.740735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.740699    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"]
Nov 02 13:37:00.749087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.749057    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:37:00.749268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.749116    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:37:00.749268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.749147    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:37:00.749475 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.749172    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:37:00.766009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.765971    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.766168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.766037    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpd5c\" (UniqueName: \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.766243 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.766193    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.867766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.867730    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.867766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.867775    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpd5c\" (UniqueName: \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.868055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.867854    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.868531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.868503    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.870223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.870197    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:00.893463 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:00.893435    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpd5c\" (UniqueName: \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\") pod \"maas-api-revoke-sub-54bee734-7x8f6\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:01.031514 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.031471    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:01.345405 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.345369    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"]
Nov 02 13:37:01.347066 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:37:01.347022    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1823438_7034_4b4c_9ecc_340e747fb17c.slice/crio-839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874 WatchSource:0}: Error finding container 839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874: Status 404 returned error can't find the container with id 839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874
Nov 02 13:37:01.511292 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.511261    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 13:37:01.574397 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.574365    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\" (UniqueName: \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\") pod \"3cf4f19c-d345-426e-8f70-20843296706d\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") "
Nov 02 13:37:01.574602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.574507    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\") pod \"3cf4f19c-d345-426e-8f70-20843296706d\" (UID: \"3cf4f19c-d345-426e-8f70-20843296706d\") "
Nov 02 13:37:01.575156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.575124    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume" pod "3cf4f19c-d345-426e-8f70-20843296706d" (UID: "3cf4f19c-d345-426e-8f70-20843296706d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:01.576630 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.576609    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp" pod "3cf4f19c-d345-426e-8f70-20843296706d" (UID: "3cf4f19c-d345-426e-8f70-20843296706d"). InnerVolumeSpecName "kube-api-access-ng8hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:01.644879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.644838    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6" event={"ID":"b1823438-7034-4b4c-9ecc-340e747fb17c","Type":"ContainerStarted","Data":"839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874"}
Nov 02 13:37:01.646549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.646515    4928 generic.go:356] "Generic (PLEG): container finished" podID="3cf4f19c-d345-426e-8f70-20843296706d" containerID="d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a" exitCode=0
Nov 02 13:37:01.646718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.646560    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"
Nov 02 13:37:01.646718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.646627    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" event={"ID":"3cf4f19c-d345-426e-8f70-20843296706d","Type":"ContainerDied","Data":"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"}
Nov 02 13:37:01.646718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.646672    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb" event={"ID":"3cf4f19c-d345-426e-8f70-20843296706d","Type":"ContainerDied","Data":"1998fbf7e4d50d0e61768eed73a99297f829f2ed46a31df96316fcee8b392bc8"}
Nov 02 13:37:01.646718 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.646704    4928 scope.go:122] "RemoveContainer" containerID="d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"
Nov 02 13:37:01.658395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.658373    4928 scope.go:122] "RemoveContainer" containerID="d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"
Nov 02 13:37:01.658667 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:01.658641    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a\": container with ID starting with d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a not found: ID does not exist" containerID="d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"
Nov 02 13:37:01.658771 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.658672    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a"} err="failed to get container status \"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a\": rpc error: code = NotFound desc = could not find container \"d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a\": container with ID starting with d02482b49864bada1efe45d8ae4f244872ea7c96c46d7e31add3c6b76db6bd2a not found: ID does not exist"
Nov 02 13:37:01.676321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.676294    4928 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf4f19c-d345-426e-8f70-20843296706d-plugins-config-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:01.676321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.676318    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8hp\" (UniqueName: \"kubernetes.io/projected/3cf4f19c-d345-426e-8f70-20843296706d-kube-api-access-ng8hp\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:01.704357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.704331    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 13:37:01.711056 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:01.711031    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8032f3-f5b5684f9-42mvb"]
Nov 02 13:37:02.497659 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:02.497611    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf4f19c-d345-426e-8f70-20843296706d" path="/var/lib/kubelet/pods/3cf4f19c-d345-426e-8f70-20843296706d/volumes"
Nov 02 13:37:02.655051 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:02.654954    4928 generic.go:356] "Generic (PLEG): container finished" podID="b1823438-7034-4b4c-9ecc-340e747fb17c" containerID="2d5db3aa567fe9f3af2d225078bc87c3fc55919b9cda28ea722ca6107dbe3d37" exitCode=0
Nov 02 13:37:02.655051 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:02.655034    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6" event={"ID":"b1823438-7034-4b4c-9ecc-340e747fb17c","Type":"ContainerDied","Data":"2d5db3aa567fe9f3af2d225078bc87c3fc55919b9cda28ea722ca6107dbe3d37"}
Nov 02 13:37:03.799005 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.798977    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:03.906283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.906248    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\") pod \"b1823438-7034-4b4c-9ecc-340e747fb17c\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") "
Nov 02 13:37:03.906461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.906434    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\") pod \"b1823438-7034-4b4c-9ecc-340e747fb17c\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") "
Nov 02 13:37:03.906512 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.906462    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\" (UniqueName: \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\") pod \"b1823438-7034-4b4c-9ecc-340e747fb17c\" (UID: \"b1823438-7034-4b4c-9ecc-340e747fb17c\") "
Nov 02 13:37:03.906627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.906601    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca" pod "b1823438-7034-4b4c-9ecc-340e747fb17c" (UID: "b1823438-7034-4b4c-9ecc-340e747fb17c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:03.906755 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.906740    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:03.908639 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.908609    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca" pod "b1823438-7034-4b4c-9ecc-340e747fb17c" (UID: "b1823438-7034-4b4c-9ecc-340e747fb17c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:03.908740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:03.908703    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c" pod "b1823438-7034-4b4c-9ecc-340e747fb17c" (UID: "b1823438-7034-4b4c-9ecc-340e747fb17c"). InnerVolumeSpecName "kube-api-access-bpd5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:04.007497 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:04.007461    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1823438-7034-4b4c-9ecc-340e747fb17c-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:04.007497 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:04.007494    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpd5c\" (UniqueName: \"kubernetes.io/projected/b1823438-7034-4b4c-9ecc-340e747fb17c-kube-api-access-bpd5c\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:04.668869 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:04.668814    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"
Nov 02 13:37:04.668869 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:04.668860    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6" event={"ID":"b1823438-7034-4b4c-9ecc-340e747fb17c","Type":"ContainerDied","Data":"839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874"}
Nov 02 13:37:04.669143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:04.668916    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839b6c680c343ea066021d511fb83f539673650eef584a247f97291476555874"
Nov 02 13:37:05.051020 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.050993    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 13:37:05.118441 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118364    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118441 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118421    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118456    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118513    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118563    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118594    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118637    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118682    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118680    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:05.118981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118741    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\") pod \"56c28cfc-c42a-49f3-80c4-a3df4be80560\" (UID: \"56c28cfc-c42a-49f3-80c4-a3df4be80560\") "
Nov 02 13:37:05.118981 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.118888    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:05.119209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.119182    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:05.119270 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.119236    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:05.119270 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.119238    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.119388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.119276    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.119878 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.119841    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:05.121222 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.121199    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:37:05.121411 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.121392    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "kube-api-access-hzkkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:05.121482 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.121461    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:05.121767 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.121749    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy" pod "56c28cfc-c42a-49f3-80c4-a3df4be80560" (UID: "56c28cfc-c42a-49f3-80c4-a3df4be80560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:05.220390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220354    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220383    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkkq\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-kube-api-access-hzkkq\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220394    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c28cfc-c42a-49f3-80c4-a3df4be80560-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220402    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220410    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c28cfc-c42a-49f3-80c4-a3df4be80560-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220427    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.220627 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.220436    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c28cfc-c42a-49f3-80c4-a3df4be80560-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:05.256589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.256556    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 13:37:05.275894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.275865    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 13:37:05.685011 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.684965    4928 generic.go:356] "Generic (PLEG): container finished" podID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerID="6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c" exitCode=0
Nov 02 13:37:05.685531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.685025    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerDied","Data":"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"}
Nov 02 13:37:05.685531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.685083    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89" event={"ID":"56c28cfc-c42a-49f3-80c4-a3df4be80560","Type":"ContainerDied","Data":"2e8d8f35d7d37916894cb8554ea77358f2b2837f28f10ed3f16eb1acdaf60feb"}
Nov 02 13:37:05.685531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.685107    4928 scope.go:122] "RemoveContainer" containerID="6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"
Nov 02 13:37:05.685531 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.685032    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"
Nov 02 13:37:05.698191 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.698170    4928 scope.go:122] "RemoveContainer" containerID="e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"
Nov 02 13:37:05.707616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.707599    4928 scope.go:122] "RemoveContainer" containerID="6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"
Nov 02 13:37:05.707895 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:05.707873    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c\": container with ID starting with 6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c not found: ID does not exist" containerID="6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"
Nov 02 13:37:05.707973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.707908    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c"} err="failed to get container status \"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c\": rpc error: code = NotFound desc = could not find container \"6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c\": container with ID starting with 6bb724ab2d627108086eaa55c09cccf312f35168465e77bc251e9f33b6d2e33c not found: ID does not exist"
Nov 02 13:37:05.707973 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.707933    4928 scope.go:122] "RemoveContainer" containerID="e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"
Nov 02 13:37:05.708201 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:05.708182    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780\": container with ID starting with e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780 not found: ID does not exist" containerID="e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"
Nov 02 13:37:05.708252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.708208    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780"} err="failed to get container status \"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780\": rpc error: code = NotFound desc = could not find container \"e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780\": container with ID starting with e73b8f9b774d14f05c5dc33998c36d6c00cda515f34e80033c5b4494d3d9c780 not found: ID does not exist"
Nov 02 13:37:05.741847 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.741809    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 13:37:05.746238 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:05.746216    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43be82-openshift-default-5df5fcbcb5-tmh89"]
Nov 02 13:37:06.498320 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:06.498280    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" path="/var/lib/kubelet/pods/56c28cfc-c42a-49f3-80c4-a3df4be80560/volumes"
Nov 02 13:37:09.729172 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:09.729130    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:37:11.637335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637298    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"]
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637715    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637727    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637748    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b1823438-7034-4b4c-9ecc-340e747fb17c" containerName="revoke-keys"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637753    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1823438-7034-4b4c-9ecc-340e747fb17c" containerName="revoke-keys"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637772    4928 cpu_manager.go:414] "containerMap: removing container" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637777    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637784    4928 cpu_manager.go:414] "containerMap: removing container" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.637791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637788    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.638098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637870    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf4f19c-d345-426e-8f70-20843296706d" containerName="payload-pre-processing"
Nov 02 13:37:11.638098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637881    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1823438-7034-4b4c-9ecc-340e747fb17c" containerName="revoke-keys"
Nov 02 13:37:11.638098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637889    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.638098 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.637895    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c28cfc-c42a-49f3-80c4-a3df4be80560" containerName="istio-proxy"
Nov 02 13:37:11.642740 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.642721    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.659933 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.659901    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:37:11.673461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.673429    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:37:11.673461 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.673429    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:37:11.673695 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.673432    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:37:11.676326 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.676300    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"]
Nov 02 13:37:11.791561 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.791496    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qd9z\" (UniqueName: \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.791800 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.791659    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.791800 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.791712    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.892957 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.892818    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.892957 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.892927    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.893218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.892987    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qd9z\" (UniqueName: \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.893648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.893615    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.895203 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.895178    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.918233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.918187    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qd9z\" (UniqueName: \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\") pod \"maas-api-revoke-sub-c7cde24e-sl6dk\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:11.956245 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:11.956210    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:12.263337 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:12.263304    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"]
Nov 02 13:37:12.267483 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:37:12.267450    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cd7b21_1829_4554_92fd_edc4b4d8bdde.slice/crio-a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc WatchSource:0}: Error finding container a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc: Status 404 returned error can't find the container with id a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc
Nov 02 13:37:12.734716 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:12.734672    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk" event={"ID":"10cd7b21-1829-4554-92fd-edc4b4d8bdde","Type":"ContainerStarted","Data":"6483b934bcd65272ff7f5d7dad78239615acbe8d3b79f1a757ed9e937bc6e468"}
Nov 02 13:37:12.735248 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:12.734736    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk" event={"ID":"10cd7b21-1829-4554-92fd-edc4b4d8bdde","Type":"ContainerStarted","Data":"a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc"}
Nov 02 13:37:12.779731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:12.779671    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk" podStartSLOduration=1.779656492 podStartE2EDuration="1.779656492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:37:12.777151377 +0000 UTC m=+4230.426205332" watchObservedRunningTime="2026-11-02 13:37:12.779656492 +0000 UTC m=+4230.428710451"
Nov 02 13:37:13.742067 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:13.742028    4928 generic.go:356] "Generic (PLEG): container finished" podID="10cd7b21-1829-4554-92fd-edc4b4d8bdde" containerID="6483b934bcd65272ff7f5d7dad78239615acbe8d3b79f1a757ed9e937bc6e468" exitCode=0
Nov 02 13:37:13.742475 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:13.742113    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk" event={"ID":"10cd7b21-1829-4554-92fd-edc4b4d8bdde","Type":"ContainerDied","Data":"6483b934bcd65272ff7f5d7dad78239615acbe8d3b79f1a757ed9e937bc6e468"}
Nov 02 13:37:14.898383 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:14.898356    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:15.024818 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.024741    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\") pod \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") "
Nov 02 13:37:15.025025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.024881    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\" (UniqueName: \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\") pod \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") "
Nov 02 13:37:15.025025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.024915    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\") pod \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\" (UID: \"10cd7b21-1829-4554-92fd-edc4b4d8bdde\") "
Nov 02 13:37:15.025214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.025184    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca" pod "10cd7b21-1829-4554-92fd-edc4b4d8bdde" (UID: "10cd7b21-1829-4554-92fd-edc4b4d8bdde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:15.027219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.027179    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca" pod "10cd7b21-1829-4554-92fd-edc4b4d8bdde" (UID: "10cd7b21-1829-4554-92fd-edc4b4d8bdde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:15.027369 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.027239    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z" pod "10cd7b21-1829-4554-92fd-edc4b4d8bdde" (UID: "10cd7b21-1829-4554-92fd-edc4b4d8bdde"). InnerVolumeSpecName "kube-api-access-2qd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:15.125803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.125762    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:15.125803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.125792    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qd9z\" (UniqueName: \"kubernetes.io/projected/10cd7b21-1829-4554-92fd-edc4b4d8bdde-kube-api-access-2qd9z\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:15.125803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.125803    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cd7b21-1829-4554-92fd-edc4b4d8bdde-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:15.754871 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.754817    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk" event={"ID":"10cd7b21-1829-4554-92fd-edc4b4d8bdde","Type":"ContainerDied","Data":"a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc"}
Nov 02 13:37:15.755093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.754884    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a743db00bd614d606de800a78d98b8e761eec1289598b97a443aa62b8bb6cecc"
Nov 02 13:37:15.755093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:15.754906    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"
Nov 02 13:37:20.277222 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.277176    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main" containerID="cri-o://2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1" gracePeriod=45
Nov 02 13:37:20.295121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.295066    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main" containerID="cri-o://87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e" gracePeriod=45
Nov 02 13:37:20.567118 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.567085    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 13:37:20.600185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.600157    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 13:37:20.682660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682617    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.682879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682677    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.682879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682718    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\" (UniqueName: \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.682879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682777    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.682879 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682853    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.683036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682890    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.683036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682926    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\" (UniqueName: \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.683036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682935    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.683036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.682970    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.683036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683018    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.683242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683064    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.683242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683092    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\") pod \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\" (UID: \"b351b1bd-4af8-4234-bd41-fa92e753e8ac\") "
Nov 02 13:37:20.683242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683095    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.683242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683123    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\") pod \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\" (UID: \"e9e0a2e6-1818-48a5-84c2-1d69cb88784d\") "
Nov 02 13:37:20.683371 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683289    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.683515 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683494    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.683720 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683704    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.683855 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683841    4928 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-model-cache\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.683977 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683712    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.684109 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683879    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.684230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.683913    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.686087 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.686045    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.686393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.686366    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "kube-api-access-n5k7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:20.686485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.686404    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.686485 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.686415    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "kube-api-access-8z9wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:20.686656 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.686618    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs" pod "b351b1bd-4af8-4234-bd41-fa92e753e8ac" (UID: "b351b1bd-4af8-4234-bd41-fa92e753e8ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:20.688282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.688260    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs" pod "e9e0a2e6-1818-48a5-84c2-1d69cb88784d" (UID: "e9e0a2e6-1818-48a5-84c2-1d69cb88784d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785601    4928 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tmp-dir\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785656    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5k7s\" (UniqueName: \"kubernetes.io/projected/b351b1bd-4af8-4234-bd41-fa92e753e8ac-kube-api-access-n5k7s\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785672    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b351b1bd-4af8-4234-bd41-fa92e753e8ac-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785686    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785699    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785712    4928 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-tls-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785725    4928 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b351b1bd-4af8-4234-bd41-fa92e753e8ac-home\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785738    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9wv\" (UniqueName: \"kubernetes.io/projected/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-kube-api-access-8z9wv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.785778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.785751    4928 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e0a2e6-1818-48a5-84c2-1d69cb88784d-dshm\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.790140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.790097    4928 generic.go:356] "Generic (PLEG): container finished" podID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerID="2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1" exitCode=0
Nov 02 13:37:20.790325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.790156    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"
Nov 02 13:37:20.790325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.790181    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" event={"ID":"e9e0a2e6-1818-48a5-84c2-1d69cb88784d","Type":"ContainerDied","Data":"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"}
Nov 02 13:37:20.790325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.790243    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz" event={"ID":"e9e0a2e6-1818-48a5-84c2-1d69cb88784d","Type":"ContainerDied","Data":"d446ee593b993fdf16f6bbae296c63409f5ed9e9d79f64d5c998cccd942402e3"}
Nov 02 13:37:20.790325 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.790266    4928 scope.go:122] "RemoveContainer" containerID="2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"
Nov 02 13:37:20.792282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.792215    4928 generic.go:356] "Generic (PLEG): container finished" podID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerID="87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e" exitCode=0
Nov 02 13:37:20.792282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.792262    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" event={"ID":"b351b1bd-4af8-4234-bd41-fa92e753e8ac","Type":"ContainerDied","Data":"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"}
Nov 02 13:37:20.792282 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.792269    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"
Nov 02 13:37:20.792548 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.792289    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw" event={"ID":"b351b1bd-4af8-4234-bd41-fa92e753e8ac","Type":"ContainerDied","Data":"02bfc8b65eb3fd25b53d379f9cf0c5ef73db2efa1b3ea0d6a565e183095e4b0d"}
Nov 02 13:37:20.802159 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.802137    4928 scope.go:122] "RemoveContainer" containerID="2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"
Nov 02 13:37:20.802466 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:20.802434    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1\": container with ID starting with 2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1 not found: ID does not exist" containerID="2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"
Nov 02 13:37:20.802554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.802465    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1"} err="failed to get container status \"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1\": rpc error: code = NotFound desc = could not find container \"2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1\": container with ID starting with 2dcd6fb4d1281296cdeb3a970eff5589310babe7d2aef7302345f7e70d1e60d1 not found: ID does not exist"
Nov 02 13:37:20.802554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.802488    4928 scope.go:122] "RemoveContainer" containerID="87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"
Nov 02 13:37:20.811167 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.811150    4928 scope.go:122] "RemoveContainer" containerID="87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"
Nov 02 13:37:20.811432 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:20.811413    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e\": container with ID starting with 87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e not found: ID does not exist" containerID="87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"
Nov 02 13:37:20.811494 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.811439    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e"} err="failed to get container status \"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e\": rpc error: code = NotFound desc = could not find container \"87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e\": container with ID starting with 87adb128652bc917e73434a9393a02a0755c09cd8862be4ef1e0b65831676b1e not found: ID does not exist"
Nov 02 13:37:20.844978 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.844949    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 13:37:20.849710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.849682    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/simulator-w0-5514e5-kserve-548559875b-j2fxw"]
Nov 02 13:37:20.884618 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.884591    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 13:37:20.888529 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:20.888504    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-w0-5514e5-kserve-5ddc4c7788-m82wz"]
Nov 02 13:37:22.496961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:22.496926    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" path="/var/lib/kubelet/pods/b351b1bd-4af8-4234-bd41-fa92e753e8ac/volumes"
Nov 02 13:37:22.497376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:22.497361    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" path="/var/lib/kubelet/pods/e9e0a2e6-1818-48a5-84c2-1d69cb88784d/volumes"
Nov 02 13:37:30.838629 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:30.838590    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:37:30.839259 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:30.838848    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy" containerID="cri-o://ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe" gracePeriod=30
Nov 02 13:37:36.096762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.096735    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:37:36.129119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129056    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129103    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129168    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129202    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129307 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129261    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129311    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129324    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:36.129489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129341    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129428    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129458    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\") pod \"ff606135-ca03-429b-9023-69a7976bee38\" (UID: \"ff606135-ca03-429b-9023-69a7976bee38\") "
Nov 02 13:37:36.129762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129570    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:36.129762 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129579    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:36.129994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129788    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:36.129994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129939    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.129994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129962    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.129994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129976    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff606135-ca03-429b-9023-69a7976bee38-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.129994 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.129989    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.130218 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.130005    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:36.131987 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.131963    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:36.132088 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.131989    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:37:36.132157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.132116    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "kube-api-access-9vvkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:36.132157 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.132128    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token" pod "ff606135-ca03-429b-9023-69a7976bee38" (UID: "ff606135-ca03-429b-9023-69a7976bee38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:36.230939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.230895    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff606135-ca03-429b-9023-69a7976bee38-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.230939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.230935    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.230939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.230946    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvkw\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-kube-api-access-9vvkw\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.230939 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.230957    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff606135-ca03-429b-9023-69a7976bee38-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.231219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.230966    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff606135-ca03-429b-9023-69a7976bee38-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:37:36.893252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.893155    4928 generic.go:356] "Generic (PLEG): container finished" podID="ff606135-ca03-429b-9023-69a7976bee38" containerID="ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe" exitCode=0
Nov 02 13:37:36.893252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.893204    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"
Nov 02 13:37:36.893252 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.893238    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" event={"ID":"ff606135-ca03-429b-9023-69a7976bee38","Type":"ContainerDied","Data":"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"}
Nov 02 13:37:36.893507 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.893290    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4" event={"ID":"ff606135-ca03-429b-9023-69a7976bee38","Type":"ContainerDied","Data":"3c8d2bbcae7784de062ae581617930fda1a2a917075131078a48865709be108c"}
Nov 02 13:37:36.893507 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.893322    4928 scope.go:122] "RemoveContainer" containerID="ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"
Nov 02 13:37:36.906124 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.906098    4928 scope.go:122] "RemoveContainer" containerID="ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"
Nov 02 13:37:36.906450 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:36.906426    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe\": container with ID starting with ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe not found: ID does not exist" containerID="ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"
Nov 02 13:37:36.906551 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.906458    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe"} err="failed to get container status \"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe\": rpc error: code = NotFound desc = could not find container \"ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe\": container with ID starting with ff2b9f6ba4b150cc6b58dc1a7907c75b5ef5249b083c3487f6fc2fade3f608fe not found: ID does not exist"
Nov 02 13:37:36.963559 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.963524    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:37:36.976985 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:36.976959    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rhhl4"]
Nov 02 13:37:38.495735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:38.495699    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff606135-ca03-429b-9023-69a7976bee38" path="/var/lib/kubelet/pods/ff606135-ca03-429b-9023-69a7976bee38/volumes"
Nov 02 13:37:39.728097 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:37:39.728058    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:37:47.559410 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:47.559361    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b88v5_fbd6770d-95d5-48fb-8fd2-c17cf1672c78/manager/0.log"
Nov 02 13:37:47.758853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:47.758781    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cxwbd_69b89c26-ba10-4856-8e9c-d13589d2ac73/manager/0.log"
Nov 02 13:37:48.417696 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:48.417659    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jpm5x_3a43c64c-0e99-4fb3-a131-042645c331f8/manager/0.log"
Nov 02 13:37:48.811141 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:37:48.811103    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-v9xwk_675b6f92-53c6-4559-9391-218bfe57c01f/manager/0.log"
Nov 02 13:38:09.728968 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:38:09.728924    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:38:15.230235 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230198    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"]
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230594    4928 cpu_manager.go:414] "containerMap: removing container" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230605    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230623    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230629    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230642    4928 cpu_manager.go:414] "containerMap: removing container" podUID="10cd7b21-1829-4554-92fd-edc4b4d8bdde" containerName="revoke-keys"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230648    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cd7b21-1829-4554-92fd-edc4b4d8bdde" containerName="revoke-keys"
Nov 02 13:38:15.230660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230663    4928 cpu_manager.go:414] "containerMap: removing container" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy"
Nov 02 13:38:15.230940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230668    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy"
Nov 02 13:38:15.230940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230725    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e0a2e6-1818-48a5-84c2-1d69cb88784d" containerName="main"
Nov 02 13:38:15.230940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230734    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cd7b21-1829-4554-92fd-edc4b4d8bdde" containerName="revoke-keys"
Nov 02 13:38:15.230940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230743    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff606135-ca03-429b-9023-69a7976bee38" containerName="istio-proxy"
Nov 02 13:38:15.230940 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.230751    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="b351b1bd-4af8-4234-bd41-fa92e753e8ac" containerName="main"
Nov 02 13:38:15.233985 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.233966    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.255617 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.255591    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"]
Nov 02 13:38:15.265986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.265955    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:38:15.266121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.265983    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:38:15.266121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.266001    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:38:15.266121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.266077    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:38:15.300952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.300923    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.301077 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.300988    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.301077 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.301058    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.401937 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.401899    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.402152 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.401987    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.402152 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.402015    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.402616 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.402578    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.404358 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:15.404332    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:15.432661 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:38:15.432641    4928 projected.go:196] Error preparing data for projected volume kube-api-access-89w2r for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 13:38:15.432764 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:38:15.432697    4928 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r podName:668914b8-102c-4d69-ba32-14792c67088e nodeName:}" failed. No retries permitted until 2026-11-02 13:38:15.932683178 +0000 UTC m=+4293.581737127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89w2r" (UniqueName: "kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r") pod "maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" (UID: "668914b8-102c-4d69-ba32-14792c67088e") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 13:38:16.006449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:16.006403    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:16.031183 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:16.031148    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") pod \"maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:16.148276 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:16.148239    4928 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" secret="" err="secret \"maas-api-cleanup-dockercfg-w9cwp\" not found"
Nov 02 13:38:16.148452 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:16.148295    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:16.451659 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:16.451634    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"]
Nov 02 13:38:16.453711 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:38:16.453680    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668914b8_102c_4d69_ba32_14792c67088e.slice/crio-21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679 WatchSource:0}: Error finding container 21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679: Status 404 returned error can't find the container with id 21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679
Nov 02 13:38:17.127723 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:17.127686    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" event={"ID":"668914b8-102c-4d69-ba32-14792c67088e","Type":"ContainerStarted","Data":"a5ed9a8e465e16f413664648c7535952285ca2057232dcba47c30a7ceaaf77cb"}
Nov 02 13:38:17.127943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:17.127740    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" event={"ID":"668914b8-102c-4d69-ba32-14792c67088e","Type":"ContainerStarted","Data":"21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679"}
Nov 02 13:38:17.128048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:17.128027    4928 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" secret="" err="secret \"maas-api-cleanup-dockercfg-w9cwp\" not found"
Nov 02 13:38:17.174323 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:17.174260    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" podStartSLOduration=2.174242013 podStartE2EDuration="2.174242013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:17.172978641 +0000 UTC m=+4294.822032598" watchObservedRunningTime="2026-11-02 13:38:17.174242013 +0000 UTC m=+4294.823295974"
Nov 02 13:38:18.136544 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:18.136506    4928 generic.go:356] "Generic (PLEG): container finished" podID="668914b8-102c-4d69-ba32-14792c67088e" containerID="a5ed9a8e465e16f413664648c7535952285ca2057232dcba47c30a7ceaaf77cb" exitCode=0
Nov 02 13:38:18.136995 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:18.136589    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" event={"ID":"668914b8-102c-4d69-ba32-14792c67088e","Type":"ContainerDied","Data":"a5ed9a8e465e16f413664648c7535952285ca2057232dcba47c30a7ceaaf77cb"}
Nov 02 13:38:19.272006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.271976    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:19.338949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.338911    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\") pod \"668914b8-102c-4d69-ba32-14792c67088e\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") "
Nov 02 13:38:19.339131 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.339006    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") pod \"668914b8-102c-4d69-ba32-14792c67088e\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") "
Nov 02 13:38:19.339131 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.339080    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\") pod \"668914b8-102c-4d69-ba32-14792c67088e\" (UID: \"668914b8-102c-4d69-ba32-14792c67088e\") "
Nov 02 13:38:19.339812 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.339777    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca" pod "668914b8-102c-4d69-ba32-14792c67088e" (UID: "668914b8-102c-4d69-ba32-14792c67088e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:19.341262 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.341236    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca" pod "668914b8-102c-4d69-ba32-14792c67088e" (UID: "668914b8-102c-4d69-ba32-14792c67088e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:19.341392 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.341271    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r" pod "668914b8-102c-4d69-ba32-14792c67088e" (UID: "668914b8-102c-4d69-ba32-14792c67088e"). InnerVolumeSpecName "kube-api-access-89w2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:19.439993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.439910    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668914b8-102c-4d69-ba32-14792c67088e-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:38:19.439993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.439941    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w2r\" (UniqueName: \"kubernetes.io/projected/668914b8-102c-4d69-ba32-14792c67088e-kube-api-access-89w2r\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:38:19.439993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:19.439952    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668914b8-102c-4d69-ba32-14792c67088e-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:38:20.148752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:20.148717    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5" event={"ID":"668914b8-102c-4d69-ba32-14792c67088e","Type":"ContainerDied","Data":"21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679"}
Nov 02 13:38:20.148752 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:20.148752    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"
Nov 02 13:38:20.148997 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:20.148767    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21049becb1f83b638b41108a71d6249c486f06b76ef906c27ae61d95aab4c679"
Nov 02 13:38:28.632982 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:28.632942    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"]
Nov 02 13:38:28.647899 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:28.647858    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-60a7443a-nt6k5"]
Nov 02 13:38:30.497258 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:38:30.497226    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668914b8-102c-4d69-ba32-14792c67088e" path="/var/lib/kubelet/pods/668914b8-102c-4d69-ba32-14792c67088e/volumes"
Nov 02 13:38:39.727589 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:38:39.727551    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:39:01.161047 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.161006    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"]
Nov 02 13:39:01.161666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.161635    4928 cpu_manager.go:414] "containerMap: removing container" podUID="668914b8-102c-4d69-ba32-14792c67088e" containerName="revoke-keys"
Nov 02 13:39:01.161666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.161649    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668914b8-102c-4d69-ba32-14792c67088e" containerName="revoke-keys"
Nov 02 13:39:01.161851 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.161810    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="668914b8-102c-4d69-ba32-14792c67088e" containerName="revoke-keys"
Nov 02 13:39:01.165454 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.165435    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.177960 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.177816    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"]
Nov 02 13:39:01.182185 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.182161    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.194749 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.194674    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"]
Nov 02 13:39:01.207263 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.207230    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"]
Nov 02 13:39:01.233552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233511    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.233732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233561    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.233732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233596    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.233732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233701    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.233732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233729    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.233963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233747    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.233963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233767    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa67824-357b-4f1b-941d-4c70f5f14994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.233963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233900    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.233963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233930    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.234165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.233986    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.234165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234010    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.234165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234060    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.234165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234085    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-kube-api-access-btbx2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.234165 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234146    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.234409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234183    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.234409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234200    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.234409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234233    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.234409 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.234295    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpd9\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-kube-api-access-5fpd9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335443    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.335489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335490    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335534    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335557    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335583    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335606    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa67824-357b-4f1b-941d-4c70f5f14994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335649    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335678    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.335763 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335719    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335952    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.335993    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336014    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336047    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336068    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336079    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-kube-api-access-btbx2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336111    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336123    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336159    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336181    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336209    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336255    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336281    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpd9\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-kube-api-access-5fpd9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336329    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336442    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336716    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336754    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.336888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.336793    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.337505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.337468    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa67824-357b-4f1b-941d-4c70f5f14994-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.338787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.338761    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.338928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.338793    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.339066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.339048    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.339175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.339154    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.360929 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.360896    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.361357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.361330    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.361576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.361559    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/f215c0d5-262f-4c8b-84ef-5496f9a8944f-kube-api-access-btbx2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f7fjr\" (UID: \"f215c0d5-262f-4c8b-84ef-5496f9a8944f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.361651 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.361622    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpd9\" (UniqueName: \"kubernetes.io/projected/5fa67824-357b-4f1b-941d-4c70f5f14994-kube-api-access-5fpd9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5g6c\" (UID: \"5fa67824-357b-4f1b-941d-4c70f5f14994\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.485065 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.485031    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:01.504569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.504535    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:01.820581 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.820528    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"]
Nov 02 13:39:01.822400 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:39:01.822349    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa67824_357b_4f1b_941d_4c70f5f14994.slice/crio-eaae57424addf1863867f8a1da8e73c59ca088568b2a692098a7dcfd51f85ef7 WatchSource:0}: Error finding container eaae57424addf1863867f8a1da8e73c59ca088568b2a692098a7dcfd51f85ef7: Status 404 returned error can't find the container with id eaae57424addf1863867f8a1da8e73c59ca088568b2a692098a7dcfd51f85ef7
Nov 02 13:39:01.823439 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.823414    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"]
Nov 02 13:39:01.824188 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:39:01.824165    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf215c0d5_262f_4c8b_84ef_5496f9a8944f.slice/crio-a21419f37d1200be8c8fd7916c3c0f2d96bcb2089a2440c6fd2327f27d9fbc00 WatchSource:0}: Error finding container a21419f37d1200be8c8fd7916c3c0f2d96bcb2089a2440c6fd2327f27d9fbc00: Status 404 returned error can't find the container with id a21419f37d1200be8c8fd7916c3c0f2d96bcb2089a2440c6fd2327f27d9fbc00
Nov 02 13:39:01.824721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.824680    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:01.824924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.824753    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:01.824924 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.824791    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:01.826657 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.826622    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:01.826754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.826700    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:01.826754 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:01.826738    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:39:02.402703 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.402664    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" event={"ID":"5fa67824-357b-4f1b-941d-4c70f5f14994","Type":"ContainerStarted","Data":"0732e1d2d5b87da25bd19dc9c4e508f06f7134df17a394e522712b59079f9028"}
Nov 02 13:39:02.403198 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.402739    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" event={"ID":"5fa67824-357b-4f1b-941d-4c70f5f14994","Type":"ContainerStarted","Data":"eaae57424addf1863867f8a1da8e73c59ca088568b2a692098a7dcfd51f85ef7"}
Nov 02 13:39:02.404360 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.404333    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" event={"ID":"f215c0d5-262f-4c8b-84ef-5496f9a8944f","Type":"ContainerStarted","Data":"189c27a9efd549f422849d15c4c4bfa5b8389c96952bba008c4ae0b8d3743ef4"}
Nov 02 13:39:02.404477 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.404377    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" event={"ID":"f215c0d5-262f-4c8b-84ef-5496f9a8944f","Type":"ContainerStarted","Data":"a21419f37d1200be8c8fd7916c3c0f2d96bcb2089a2440c6fd2327f27d9fbc00"}
Nov 02 13:39:02.485922 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.485887    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:02.487230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.487203    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:02.487374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.487248    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:02.504737 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.504715    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:02.505904 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.505881    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:02.506007 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.505920    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:02.515375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.515327    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podStartSLOduration=1.515310209 podStartE2EDuration="1.515310209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:02.512608288 +0000 UTC m=+4340.161662249" watchObservedRunningTime="2026-11-02 13:39:02.515310209 +0000 UTC m=+4340.164364168"
Nov 02 13:39:02.516136 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:02.516096    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podStartSLOduration=1.516084199 podStartE2EDuration="1.516084199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:02.46287442 +0000 UTC m=+4340.111928370" watchObservedRunningTime="2026-11-02 13:39:02.516084199 +0000 UTC m=+4340.165138158"
Nov 02 13:39:03.486887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:03.486368    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:03.486887 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:03.486436    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:03.505576 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:03.505540    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:03.505735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:03.505601    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:04.486398 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:04.486350    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:04.486596 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:04.486413    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:04.504984 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:04.504953    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:04.505368 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:04.504998    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:05.485798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:05.485752    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:05.486010 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:05.485810    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:05.505949 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:05.505916    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:05.506338 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:05.505965    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:06.486926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:06.486401    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:06.486926 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:06.486476    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:06.505380 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:06.505344    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:06.505581 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:06.505404    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:07.485609 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:07.485566    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:07.486062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:07.485637    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:07.505753 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:07.505719    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:07.505910 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:07.505797    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:08.486283 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:08.486248    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:08.486779 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:08.486306    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:08.505059 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:08.505026    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:08.505234 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:08.505077    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:09.485425 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:09.485391    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 13:39:09.485598 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:09.485447    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 13:39:09.505038 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:09.505012    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:39:09.505375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:09.505061    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Nov 02 13:39:09.729483 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:39:09.729439    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:39:11.486175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:11.486117    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:11.486737 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:11.486186    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:11.547074 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:11.547022    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:11.547309 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:11.547103    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.528188 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.528073    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:12.528900 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.528850    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.589990 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.589942    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:12.590192 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.590011    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.814054 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.813962    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:12.814431 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.814404    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:12.815347 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.815325    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr"
Nov 02 13:39:12.977219 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.977187    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:12.977946 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:12.977919    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:13.044486 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:13.044455    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c"
Nov 02 13:39:34.030908 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:34.030865    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"]
Nov 02 13:39:34.044037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:34.044007    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1560a6-qds4q"]
Nov 02 13:39:34.497169 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:34.497132    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579342b9-0a69-4d1e-bdef-31becacf4f0f" path="/var/lib/kubelet/pods/579342b9-0a69-4d1e-bdef-31becacf4f0f/volumes"
Nov 02 13:39:39.728213 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:39:39.728174    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:39:45.031780 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:45.031746    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"]
Nov 02 13:39:45.045028 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:45.044994    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e70ab4-vdsm8"]
Nov 02 13:39:46.496006 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:39:46.495967    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f395794-bb0b-4e24-a343-71a2d56b7bfd" path="/var/lib/kubelet/pods/3f395794-bb0b-4e24-a343-71a2d56b7bfd/volumes"
Nov 02 13:40:09.728120 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:40:09.728075    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:40:16.034426 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:16.034376    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"]
Nov 02 13:40:16.047314 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:16.047274    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1ea3d6-95wtb"]
Nov 02 13:40:16.507858 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:16.506725    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345feb12-e066-4a6b-9f46-5006d8563049" path="/var/lib/kubelet/pods/345feb12-e066-4a6b-9f46-5006d8563049/volumes"
Nov 02 13:40:27.031156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:27.031040    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"]
Nov 02 13:40:27.044547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:27.044513    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fefdb5b-2qtmm"]
Nov 02 13:40:28.496141 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:28.496107    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114647d3-4b75-40a7-922f-0e2e9b72a797" path="/var/lib/kubelet/pods/114647d3-4b75-40a7-922f-0e2e9b72a797/volumes"
Nov 02 13:40:38.035734 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:38.035694    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"]
Nov 02 13:40:38.049667 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:38.049634    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a55b1f1-rdwm9"]
Nov 02 13:40:38.496033 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:38.495997    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7ec296-0dc8-46c0-be1b-1384813b5ce3" path="/var/lib/kubelet/pods/2c7ec296-0dc8-46c0-be1b-1384813b5ce3/volumes"
Nov 02 13:40:39.727643 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:40:39.727600    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:40:45.355242 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:45.355203    4928 scope.go:122] "RemoveContainer" containerID="996b74bbbb4d570f217f4c02fe7ed57e0dae1561e7e06b32511d282d75e27827"
Nov 02 13:40:45.369993 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:40:45.369625    4928 scope.go:122] "RemoveContainer" containerID="bdaff4e380b17593f88bff6eb4f5da32e9e1ef4e9a23c792c8bc8d791d44e783"
Nov 02 13:41:09.728051 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:41:09.728014    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:41:32.667903 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.667866    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"]
Nov 02 13:41:32.671361 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.671342    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.724431 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.724402    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:41:32.724686 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.724669    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:41:32.737868 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.737838    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:41:32.738042 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.737903    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:41:32.741115 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.741089    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"]
Nov 02 13:41:32.759378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.759343    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.759378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.759381    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ctc\" (UniqueName: \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.759583 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.759403    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.860623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.860582    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.860623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.860627    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ctc\" (UniqueName: \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.860876 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.860735    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.861266 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.861235    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.863100 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.863083    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.885668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.885638    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ctc\" (UniqueName: \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\") pod \"maas-api-revoke-sub-13cfd66b-92ms4\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:32.984121 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:32.984085    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:33.290867 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:33.290793    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"]
Nov 02 13:41:33.293108 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:41:33.293079    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b39dff4_7faf_41fa_82ce_6773154ead5a.slice/crio-51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2 WatchSource:0}: Error finding container 51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2: Status 404 returned error can't find the container with id 51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2
Nov 02 13:41:33.492717 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:33.492680    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4" event={"ID":"1b39dff4-7faf-41fa-82ce-6773154ead5a","Type":"ContainerStarted","Data":"51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2"}
Nov 02 13:41:34.500134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:34.500097    4928 generic.go:356] "Generic (PLEG): container finished" podID="1b39dff4-7faf-41fa-82ce-6773154ead5a" containerID="bf5e00d8b1e13b4a2b07b788f6fb0f3f27d9b245030518fb81fa10ad430aef8b" exitCode=0
Nov 02 13:41:34.500612 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:34.500168    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4" event={"ID":"1b39dff4-7faf-41fa-82ce-6773154ead5a","Type":"ContainerDied","Data":"bf5e00d8b1e13b4a2b07b788f6fb0f3f27d9b245030518fb81fa10ad430aef8b"}
Nov 02 13:41:35.651463 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.651429    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:35.682914 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.682883    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\") pod \"1b39dff4-7faf-41fa-82ce-6773154ead5a\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") "
Nov 02 13:41:35.683070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.682955    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\") pod \"1b39dff4-7faf-41fa-82ce-6773154ead5a\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") "
Nov 02 13:41:35.683070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.682979    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\" (UniqueName: \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\") pod \"1b39dff4-7faf-41fa-82ce-6773154ead5a\" (UID: \"1b39dff4-7faf-41fa-82ce-6773154ead5a\") "
Nov 02 13:41:35.683335 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.683310    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca" pod "1b39dff4-7faf-41fa-82ce-6773154ead5a" (UID: "1b39dff4-7faf-41fa-82ce-6773154ead5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:41:35.684987 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.684964    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca" pod "1b39dff4-7faf-41fa-82ce-6773154ead5a" (UID: "1b39dff4-7faf-41fa-82ce-6773154ead5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:41:35.685155 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.685134    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc" pod "1b39dff4-7faf-41fa-82ce-6773154ead5a" (UID: "1b39dff4-7faf-41fa-82ce-6773154ead5a"). InnerVolumeSpecName "kube-api-access-x6ctc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:41:35.784344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.784258    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:35.784344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.784294    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b39dff4-7faf-41fa-82ce-6773154ead5a-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:35.784344 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:35.784306    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ctc\" (UniqueName: \"kubernetes.io/projected/1b39dff4-7faf-41fa-82ce-6773154ead5a-kube-api-access-x6ctc\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:36.511848 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:36.511807    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"
Nov 02 13:41:36.511986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:36.511843    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4" event={"ID":"1b39dff4-7faf-41fa-82ce-6773154ead5a","Type":"ContainerDied","Data":"51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2"}
Nov 02 13:41:36.511986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:36.511899    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b3a64206250a8c638124aedeaec751436ba9f8ae8f8d255493a7c026130bd2"
Nov 02 13:41:39.728688 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:41:39.728648    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:41:43.446304 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:43.446279    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:41:43.468048 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:43.468021    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:41:45.409055 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:45.409017    4928 scope.go:122] "RemoveContainer" containerID="d51e7a9e6fc0054f1fba26f4e029f756e499c64cce7aedd5db9a49950475a93c"
Nov 02 13:41:45.418553 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:45.418530    4928 scope.go:122] "RemoveContainer" containerID="cded59bd18433e289491ebe3f51aaf4404fffcf2ef612220ad047741df4c2942"
Nov 02 13:41:45.426732 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:45.426712    4928 scope.go:122] "RemoveContainer" containerID="f6a01084da4ed883cd6434256dd631bb19dfb4a0742ec0ed04b3f20216561a4c"
Nov 02 13:41:47.234391 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.234350    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"]
Nov 02 13:41:47.234969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.234952    4928 cpu_manager.go:414] "containerMap: removing container" podUID="1b39dff4-7faf-41fa-82ce-6773154ead5a" containerName="revoke-keys"
Nov 02 13:41:47.235035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.234972    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b39dff4-7faf-41fa-82ce-6773154ead5a" containerName="revoke-keys"
Nov 02 13:41:47.235134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.235110    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b39dff4-7faf-41fa-82ce-6773154ead5a" containerName="revoke-keys"
Nov 02 13:41:47.240265 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.240235    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.275340 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.275301    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"]
Nov 02 13:41:47.282064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.282032    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.282236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.282083    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.282236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.282208    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kqk\" (UniqueName: \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.284725 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.284698    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:41:47.284894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.284737    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:41:47.284894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.284740    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:41:47.284894 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.284784    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:41:47.383551 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.383513    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.383729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.383562    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.383729 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.383614    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kqk\" (UniqueName: \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.384396 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.384369    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.386137 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.386116    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.409319 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.409285    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kqk\" (UniqueName: \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\") pod \"maas-api-revoke-sub-343fa587-9xwtl\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.556278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.556183    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:47.860024 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:47.859997    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"]
Nov 02 13:41:47.862879 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:41:47.862794    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a641d4_2e20_4162_9643_181ce2880d64.slice/crio-3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999 WatchSource:0}: Error finding container 3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999: Status 404 returned error can't find the container with id 3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999
Nov 02 13:41:48.595060 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:48.595021    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl" event={"ID":"78a641d4-2e20-4162-9643-181ce2880d64","Type":"ContainerStarted","Data":"2e4796d33845f6eaebb77e76bbc22949f68aeb63006a84f361d9f20757e5ad4a"}
Nov 02 13:41:48.595602 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:48.595077    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl" event={"ID":"78a641d4-2e20-4162-9643-181ce2880d64","Type":"ContainerStarted","Data":"3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999"}
Nov 02 13:41:48.640216 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:48.640155    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl" podStartSLOduration=1.6401392399999999 podStartE2EDuration="1.64013924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:41:48.637477892 +0000 UTC m=+4506.286531866" watchObservedRunningTime="2026-11-02 13:41:48.64013924 +0000 UTC m=+4506.289193198"
Nov 02 13:41:49.605852 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:49.605696    4928 generic.go:356] "Generic (PLEG): container finished" podID="78a641d4-2e20-4162-9643-181ce2880d64" containerID="2e4796d33845f6eaebb77e76bbc22949f68aeb63006a84f361d9f20757e5ad4a" exitCode=0
Nov 02 13:41:49.606395 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:49.606201    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl" event={"ID":"78a641d4-2e20-4162-9643-181ce2880d64","Type":"ContainerDied","Data":"2e4796d33845f6eaebb77e76bbc22949f68aeb63006a84f361d9f20757e5ad4a"}
Nov 02 13:41:50.810609 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.810580    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:50.830753 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.830721    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\" (UniqueName: \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\") pod \"78a641d4-2e20-4162-9643-181ce2880d64\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") "
Nov 02 13:41:50.830952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.830854    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\") pod \"78a641d4-2e20-4162-9643-181ce2880d64\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") "
Nov 02 13:41:50.830952 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.830901    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\") pod \"78a641d4-2e20-4162-9643-181ce2880d64\" (UID: \"78a641d4-2e20-4162-9643-181ce2880d64\") "
Nov 02 13:41:50.831378 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.831347    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca" pod "78a641d4-2e20-4162-9643-181ce2880d64" (UID: "78a641d4-2e20-4162-9643-181ce2880d64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:41:50.833140 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.833117    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca" pod "78a641d4-2e20-4162-9643-181ce2880d64" (UID: "78a641d4-2e20-4162-9643-181ce2880d64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:41:50.833401 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.833376    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk" pod "78a641d4-2e20-4162-9643-181ce2880d64" (UID: "78a641d4-2e20-4162-9643-181ce2880d64"). InnerVolumeSpecName "kube-api-access-z9kqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:41:50.931954 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.931864    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:50.931954 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.931893    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a641d4-2e20-4162-9643-181ce2880d64-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:50.931954 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:50.931904    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kqk\" (UniqueName: \"kubernetes.io/projected/78a641d4-2e20-4162-9643-181ce2880d64-kube-api-access-z9kqk\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:41:51.628163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:51.628114    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl" event={"ID":"78a641d4-2e20-4162-9643-181ce2880d64","Type":"ContainerDied","Data":"3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999"}
Nov 02 13:41:51.628375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:51.628139    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"
Nov 02 13:41:51.628375 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:51.628212    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3558812f1fd093bebac32718b1ade38f4bdb3a904a17508a65fc6c81607f4999"
Nov 02 13:41:59.594112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.594076    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:41:59.594632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.594562    4928 cpu_manager.go:414] "containerMap: removing container" podUID="78a641d4-2e20-4162-9643-181ce2880d64" containerName="revoke-keys"
Nov 02 13:41:59.594632 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.594578    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a641d4-2e20-4162-9643-181ce2880d64" containerName="revoke-keys"
Nov 02 13:41:59.594726 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.594657    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a641d4-2e20-4162-9643-181ce2880d64" containerName="revoke-keys"
Nov 02 13:41:59.597372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.597347    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.618481 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.618450    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f119d1-openshift-default-dockercfg-85dxh\""
Nov 02 13:41:59.634421 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.634389    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:41:59.650624 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650594    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650633    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650789 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650761    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650799    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650816    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650933    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282mh\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.650975 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.650973    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.651220 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.651022    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.651220 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.651083    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.751965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.751932    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.751965 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.751978    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752158    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752197    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752228 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752217    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752340 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752242    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752340 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752283    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282mh\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752336    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752449 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752374    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752552 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752473    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752634 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752603    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752705 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752605    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752747 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752717    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.752963 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.752941    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.754622 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.754594    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.754838 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.754812    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.780017 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.779984    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.780885 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.780734    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282mh\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\") pod \"e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") " pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:41:59.914743 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:41:59.914709    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:00.237176 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.237153    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:42:00.239447 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:42:00.239412    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5056eb26_8194_4f49_882c_954b4e998d77.slice/crio-39093f54d747125012889e28489e86462b3b7d0e12fa3dcd60a5e8a022bcdb5f WatchSource:0}: Error finding container 39093f54d747125012889e28489e86462b3b7d0e12fa3dcd60a5e8a022bcdb5f: Status 404 returned error can't find the container with id 39093f54d747125012889e28489e86462b3b7d0e12fa3dcd60a5e8a022bcdb5f
Nov 02 13:42:00.241785 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.241747    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:00.241928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.241861    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:00.241928 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.241914    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:00.724278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.724193    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerStarted","Data":"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"}
Nov 02 13:42:00.724278 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.724242    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerStarted","Data":"39093f54d747125012889e28489e86462b3b7d0e12fa3dcd60a5e8a022bcdb5f"}
Nov 02 13:42:00.927809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:00.927774    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:01.920633 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:01.920601    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:01.985690 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:01.985635    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podStartSLOduration=2.985620731 podStartE2EDuration="2.985620731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:00.813272651 +0000 UTC m=+4518.462326609" watchObservedRunningTime="2026-11-02 13:42:01.985620731 +0000 UTC m=+4519.634674689"
Nov 02 13:42:02.736544 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:02.736509    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:02.737741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:02.737718    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:04.030589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.030547    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"]
Nov 02 13:42:04.043805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.043775    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bee734-7x8f6"]
Nov 02 13:42:04.495805 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.495767    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1823438-7034-4b4c-9ecc-340e747fb17c" path="/var/lib/kubelet/pods/b1823438-7034-4b4c-9ecc-340e747fb17c/volumes"
Nov 02 13:42:04.725046 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.725004    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:42:04.730151 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.730126    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.761802 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.761718    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f119d1-serving-cert\""
Nov 02 13:42:04.762084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.762056    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:42:04.762230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.762139    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:42:04.762297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.762227    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:42:04.762297 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.762231    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:42:04.765291 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.765266    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:42:04.804691 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.804661    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkh9\" (UniqueName: \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.804909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.804719    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.804991 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.804959    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.805214 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.805195    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.906471 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.906437    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkh9\" (UniqueName: \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.906688 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.906475    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.906688 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.906538    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.906688 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.906620    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.907394 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.907368    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.909149 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.909124    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.909289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.909166    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:04.931026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:04.930996    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkh9\" (UniqueName: \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:05.044479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:05.044386    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:05.359675 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:05.359639    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:42:05.362311 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:42:05.362283    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25d664e_4544_4a3c_883d_5c3fcf54ef0b.slice/crio-988e1f3d2f865c270c949eb256daec735b8190962cf3f704a08f63b2adadb65d WatchSource:0}: Error finding container 988e1f3d2f865c270c949eb256daec735b8190962cf3f704a08f63b2adadb65d: Status 404 returned error can't find the container with id 988e1f3d2f865c270c949eb256daec735b8190962cf3f704a08f63b2adadb65d
Nov 02 13:42:05.363746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:05.363723    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:42:05.757436 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:05.757399    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" event={"ID":"c25d664e-4544-4a3c-883d-5c3fcf54ef0b","Type":"ContainerStarted","Data":"988e1f3d2f865c270c949eb256daec735b8190962cf3f704a08f63b2adadb65d"}
Nov 02 13:42:06.764886 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:06.764841    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" event={"ID":"c25d664e-4544-4a3c-883d-5c3fcf54ef0b","Type":"ContainerStarted","Data":"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"}
Nov 02 13:42:06.765289 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:06.764939    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:06.812023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:06.811964    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" podStartSLOduration=2.350191501 podStartE2EDuration="2.811948964s" totalImagesPullingTime="461.757463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:06.810971428 +0000 UTC m=+4524.460025386" watchObservedRunningTime="2026-11-02 13:42:06.811948964 +0000 UTC m=+4524.461002923"
Nov 02 13:42:09.758112 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:42:09.758073    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:42:12.784234 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:12.784203    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:42:15.035388 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:15.035333    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"]
Nov 02 13:42:15.048355 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:15.048327    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7cde24e-sl6dk"]
Nov 02 13:42:16.495922 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:16.495886    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cd7b21-1829-4554-92fd-edc4b4d8bdde" path="/var/lib/kubelet/pods/10cd7b21-1829-4554-92fd-edc4b4d8bdde/volumes"
Nov 02 13:42:17.818676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.817798    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:51302->10.132.0.238:15021: read: connection reset by peer" start-of-body=
Nov 02 13:42:17.818676 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.817893    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": read tcp 10.132.0.2:51302->10.132.0.238:15021: read: connection reset by peer"
Nov 02 13:42:17.872015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.871987    4928 generic.go:356] "Generic (PLEG): container finished" podID="5056eb26-8194-4f49-882c-954b4e998d77" containerID="3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6" exitCode=0
Nov 02 13:42:17.872153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.872058    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerDied","Data":"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"}
Nov 02 13:42:17.872468 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.872450    4928 scope.go:122] "RemoveContainer" containerID="3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"
Nov 02 13:42:17.873721 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.873694    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:17.873787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.873762    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:17.873842 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:17.873796    4928 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 13:42:18.889775 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:18.889734    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerStarted","Data":"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"}
Nov 02 13:42:18.915361 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:18.915329    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:18.915527 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:18.915471    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:18.915583 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:18.915524    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:19.916057 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:19.916014    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:19.916494 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:19.916076    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:20.916113 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:20.916073    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:20.916569 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:20.916173    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:21.915554 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:21.915518    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:21.915766 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:21.915575    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:22.915143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:22.914972    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:22.915143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:22.915020    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:23.916139 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:23.916049    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 13:42:23.916139 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:23.916123    4928 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 13:42:24.919778 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:24.919744    4928 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:24.920273 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:24.920259    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:24.921070 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:42:24.921047    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:42:39.728596 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:42:39.728556    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:43:08.583537 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:08.583502    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:43:08.584066 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:08.583762    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" podUID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" containerName="maas-api" containerID="cri-o://2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b" gracePeriod=30
Nov 02 13:43:09.099257 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.099228    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:43:09.191961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.191863    4928 generic.go:356] "Generic (PLEG): container finished" podID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" containerID="2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b" exitCode=0
Nov 02 13:43:09.191961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.191928    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"
Nov 02 13:43:09.191961 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.191924    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" event={"ID":"c25d664e-4544-4a3c-883d-5c3fcf54ef0b","Type":"ContainerDied","Data":"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"}
Nov 02 13:43:09.192236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.191992    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt" event={"ID":"c25d664e-4544-4a3c-883d-5c3fcf54ef0b","Type":"ContainerDied","Data":"988e1f3d2f865c270c949eb256daec735b8190962cf3f704a08f63b2adadb65d"}
Nov 02 13:43:09.192236 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.192024    4928 scope.go:122] "RemoveContainer" containerID="2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"
Nov 02 13:43:09.202247 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.202223    4928 scope.go:122] "RemoveContainer" containerID="2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"
Nov 02 13:43:09.202495 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:43:09.202475    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b\": container with ID starting with 2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b not found: ID does not exist" containerID="2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"
Nov 02 13:43:09.202578 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.202503    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b"} err="failed to get container status \"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b\": rpc error: code = NotFound desc = could not find container \"2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b\": container with ID starting with 2e04e17c66636b0dfbcea951c84fc9532a9f29a83855b9ced9e83f784279140b not found: ID does not exist"
Nov 02 13:43:09.210272 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210242    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\") pod \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") "
Nov 02 13:43:09.210410 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210348    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\") pod \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") "
Nov 02 13:43:09.210410 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210385    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\" (UniqueName: \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\") pod \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") "
Nov 02 13:43:09.210410 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210406    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\") pod \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\" (UID: \"c25d664e-4544-4a3c-883d-5c3fcf54ef0b\") "
Nov 02 13:43:09.210660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210637    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca" pod "c25d664e-4544-4a3c-883d-5c3fcf54ef0b" (UID: "c25d664e-4544-4a3c-883d-5c3fcf54ef0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:43:09.210916 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.210882    4928 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-openshift-service-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:09.212623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.212598    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls" pod "c25d664e-4544-4a3c-883d-5c3fcf54ef0b" (UID: "c25d664e-4544-4a3c-883d-5c3fcf54ef0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:43:09.212722 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.212684    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9" pod "c25d664e-4544-4a3c-883d-5c3fcf54ef0b" (UID: "c25d664e-4544-4a3c-883d-5c3fcf54ef0b"). InnerVolumeSpecName "kube-api-access-mnkh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:43:09.213003 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.212981    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs" pod "c25d664e-4544-4a3c-883d-5c3fcf54ef0b" (UID: "c25d664e-4544-4a3c-883d-5c3fcf54ef0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:43:09.311664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.311616    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-maas-api-tls\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:09.311664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.311658    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnkh9\" (UniqueName: \"kubernetes.io/projected/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-kube-api-access-mnkh9\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:09.311664 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.311669    4928 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c25d664e-4544-4a3c-883d-5c3fcf54ef0b-metrics-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:09.540025 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.539985    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:43:09.553205 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:09.553177    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-z88kt"]
Nov 02 13:43:09.729005 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:43:09.728968    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:43:10.497623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:10.497587    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" path="/var/lib/kubelet/pods/c25d664e-4544-4a3c-883d-5c3fcf54ef0b/volumes"
Nov 02 13:43:39.728302 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:43:39.728266    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:43:45.464872 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:45.464817    4928 scope.go:122] "RemoveContainer" containerID="2d5db3aa567fe9f3af2d225078bc87c3fc55919b9cda28ea722ca6107dbe3d37"
Nov 02 13:43:45.474230 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:45.474210    4928 scope.go:122] "RemoveContainer" containerID="6483b934bcd65272ff7f5d7dad78239615acbe8d3b79f1a757ed9e937bc6e468"
Nov 02 13:43:52.168505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.168129    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"]
Nov 02 13:43:52.183513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.178672    4928 cpu_manager.go:414] "containerMap: removing container" podUID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" containerName="maas-api"
Nov 02 13:43:52.183513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.178704    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" containerName="maas-api"
Nov 02 13:43:52.183513 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.178867    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25d664e-4544-4a3c-883d-5c3fcf54ef0b" containerName="maas-api"
Nov 02 13:43:52.184553 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.184415    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.203153 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.203117    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"]
Nov 02 13:43:52.219374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.218578    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:43:52.219374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.218629    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:43:52.219374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.218736    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:43:52.219374 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.218784    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:43:52.259390 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.259358    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.259588 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.259398    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vqk\" (UniqueName: \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.259588 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.259538    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.360446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.360403    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.360446 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.360452    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vqk\" (UniqueName: \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.360665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.360532    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.361107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.361086    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.363147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.363114    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.385457 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.385427    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vqk\" (UniqueName: \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.540348 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.540309    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:52.849102 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:52.849074    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"]
Nov 02 13:43:52.851409 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:43:52.851379    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0ab067_b839_427e_84b7_c13a67c211da.slice/crio-f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b WatchSource:0}: Error finding container f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b: Status 404 returned error can't find the container with id f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b
Nov 02 13:43:53.512530 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:53.512493    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct" event={"ID":"5d0ab067-b839-427e-84b7-c13a67c211da","Type":"ContainerStarted","Data":"39299a6ccf206d487fde5f6970083035f82072bb6fe3e4d7b93bee2a178f9783"}
Nov 02 13:43:53.512968 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:53.512554    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct" event={"ID":"5d0ab067-b839-427e-84b7-c13a67c211da","Type":"ContainerStarted","Data":"f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b"}
Nov 02 13:43:53.560093 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:53.560042    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct" podStartSLOduration=1.560029021 podStartE2EDuration="1.560029021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:53.556962932 +0000 UTC m=+4631.206016901" watchObservedRunningTime="2026-11-02 13:43:53.560029021 +0000 UTC m=+4631.209083037"
Nov 02 13:43:54.520064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:54.520030    4928 generic.go:356] "Generic (PLEG): container finished" podID="5d0ab067-b839-427e-84b7-c13a67c211da" containerID="39299a6ccf206d487fde5f6970083035f82072bb6fe3e4d7b93bee2a178f9783" exitCode=0
Nov 02 13:43:54.520064 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:54.520067    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct" event={"ID":"5d0ab067-b839-427e-84b7-c13a67c211da","Type":"ContainerDied","Data":"39299a6ccf206d487fde5f6970083035f82072bb6fe3e4d7b93bee2a178f9783"}
Nov 02 13:43:55.675178 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.675149    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:55.691931 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.691897    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\") pod \"5d0ab067-b839-427e-84b7-c13a67c211da\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") "
Nov 02 13:43:55.692076 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.691983    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\") pod \"5d0ab067-b839-427e-84b7-c13a67c211da\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") "
Nov 02 13:43:55.692128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.692105    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\" (UniqueName: \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\") pod \"5d0ab067-b839-427e-84b7-c13a67c211da\" (UID: \"5d0ab067-b839-427e-84b7-c13a67c211da\") "
Nov 02 13:43:55.692212 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.692190    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca" pod "5d0ab067-b839-427e-84b7-c13a67c211da" (UID: "5d0ab067-b839-427e-84b7-c13a67c211da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:43:55.692434 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.692412    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:55.694312 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.694288    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk" pod "5d0ab067-b839-427e-84b7-c13a67c211da" (UID: "5d0ab067-b839-427e-84b7-c13a67c211da"). InnerVolumeSpecName "kube-api-access-p7vqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:43:55.695152 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.695131    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca" pod "5d0ab067-b839-427e-84b7-c13a67c211da" (UID: "5d0ab067-b839-427e-84b7-c13a67c211da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:43:55.794023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.793936    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vqk\" (UniqueName: \"kubernetes.io/projected/5d0ab067-b839-427e-84b7-c13a67c211da-kube-api-access-p7vqk\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:55.794023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:55.793970    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0ab067-b839-427e-84b7-c13a67c211da-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:43:56.532112 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:56.532079    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"
Nov 02 13:43:56.532287 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:56.532115    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct" event={"ID":"5d0ab067-b839-427e-84b7-c13a67c211da","Type":"ContainerDied","Data":"f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b"}
Nov 02 13:43:56.532287 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:43:56.532178    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9aecf61c3f96edc1f851494eaec1f62c0fbccb93aa86045a67f6f4b321f121b"
Nov 02 13:44:05.489562 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:05.489522    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"]
Nov 02 13:44:05.502728 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:05.502700    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f119d1-pnnct"]
Nov 02 13:44:05.847589 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:05.847498    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:44:05.847912 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:05.847855    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" containerID="cri-o://b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26" gracePeriod=30
Nov 02 13:44:06.496144 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:06.496098    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0ab067-b839-427e-84b7-c13a67c211da" path="/var/lib/kubelet/pods/5d0ab067-b839-427e-84b7-c13a67c211da/volumes"
Nov 02 13:44:09.728978 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:44:09.728935    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:44:09.923168 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:09.923131    4928 patch_prober.go:29] interesting pod/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:44:09.923336 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:09.923191    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:44:11.513495 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.513466    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:44:11.618662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.618571    4928 generic.go:356] "Generic (PLEG): container finished" podID="5056eb26-8194-4f49-882c-954b4e998d77" containerID="b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26" exitCode=0
Nov 02 13:44:11.618662 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.618621    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"
Nov 02 13:44:11.618955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.618659    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerDied","Data":"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"}
Nov 02 13:44:11.618955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.618724    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk" event={"ID":"5056eb26-8194-4f49-882c-954b4e998d77","Type":"ContainerDied","Data":"39093f54d747125012889e28489e86462b3b7d0e12fa3dcd60a5e8a022bcdb5f"}
Nov 02 13:44:11.618955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.618756    4928 scope.go:122] "RemoveContainer" containerID="b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"
Nov 02 13:44:11.628163 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.628144    4928 scope.go:122] "RemoveContainer" containerID="3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"
Nov 02 13:44:11.641920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.641895    4928 scope.go:122] "RemoveContainer" containerID="b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"
Nov 02 13:44:11.642391 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:44:11.642363    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26\": container with ID starting with b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26 not found: ID does not exist" containerID="b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"
Nov 02 13:44:11.642517 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.642400    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26"} err="failed to get container status \"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26\": rpc error: code = NotFound desc = could not find container \"b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26\": container with ID starting with b0d461edac54b5b39ebbd045c42999a195c01abb86ac1446acbd48bbfb1d6e26 not found: ID does not exist"
Nov 02 13:44:11.642517 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.642425    4928 scope.go:122] "RemoveContainer" containerID="3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"
Nov 02 13:44:11.642755 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:44:11.642728    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6\": container with ID starting with 3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6 not found: ID does not exist" containerID="3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"
Nov 02 13:44:11.642809 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.642764    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6"} err="failed to get container status \"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6\": rpc error: code = NotFound desc = could not find container \"3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6\": container with ID starting with 3ad22ccdcbbe9ba6b57188749706d05904dd8b3ee64ba9fee4d37948f3df71e6 not found: ID does not exist"
Nov 02 13:44:11.655166 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655139    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655285 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655237    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655285 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655273    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655411 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655337    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655411 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655376    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655448    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655486    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655532 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655519    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655533    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.655710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655559    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\") pod \"5056eb26-8194-4f49-882c-954b4e998d77\" (UID: \"5056eb26-8194-4f49-882c-954b4e998d77\") "
Nov 02 13:44:11.655710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655650    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.655710 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655665    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.656213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.655988    4928 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-certs\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.656213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.656011    4928 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-credential-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.656213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.656026    4928 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-workload-socket\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.656213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.656022    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:11.656213 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.656177    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.657798 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.657777    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.657938 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.657894    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:44:11.658180 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.658156    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "kube-api-access-282mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:11.658389 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.658363    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token" pod "5056eb26-8194-4f49-882c-954b4e998d77" (UID: "5056eb26-8194-4f49-882c-954b4e998d77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:11.756615 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756580    4928 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-data\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.756615 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756607    4928 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5056eb26-8194-4f49-882c-954b4e998d77-istio-podinfo\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.756615 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756619    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282mh\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-kube-api-access-282mh\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.756615 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756628    4928 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5056eb26-8194-4f49-882c-954b4e998d77-istiod-ca-cert\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.756943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756636    4928 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5056eb26-8194-4f49-882c-954b4e998d77-istio-token\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.756943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.756644    4928 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5056eb26-8194-4f49-882c-954b4e998d77-istio-envoy\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:44:11.973677 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.973631    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:44:11.987202 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:11.987164    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f119d1-openshift-default-5868b76b8fnzjtk"]
Nov 02 13:44:12.496293 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:12.496259    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5056eb26-8194-4f49-882c-954b4e998d77" path="/var/lib/kubelet/pods/5056eb26-8194-4f49-882c-954b4e998d77/volumes"
Nov 02 13:44:39.728764 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:44:39.728724    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:44:45.507564 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:45.507530    4928 scope.go:122] "RemoveContainer" containerID="a5ed9a8e465e16f413664648c7535952285ca2057232dcba47c30a7ceaaf77cb"
Nov 02 13:44:48.396034 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:48.395981    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4g2zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:44:48.396633 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:44:48.396084    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4g2zz" podUID="ce79f7b4-4d9f-461c-bac8-5ff7576cf72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:45:03.233990 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.233953    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"]
Nov 02 13:45:03.234658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234620    4928 cpu_manager.go:414] "containerMap: removing container" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.234658 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234646    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.234898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234676    4928 cpu_manager.go:414] "containerMap: removing container" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.234898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234685    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.234898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234711    4928 cpu_manager.go:414] "containerMap: removing container" podUID="5d0ab067-b839-427e-84b7-c13a67c211da" containerName="revoke-keys"
Nov 02 13:45:03.234898 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234719    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0ab067-b839-427e-84b7-c13a67c211da" containerName="revoke-keys"
Nov 02 13:45:03.235156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234917    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.235156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234940    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="5056eb26-8194-4f49-882c-954b4e998d77" containerName="istio-proxy"
Nov 02 13:45:03.235156 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.234953    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0ab067-b839-427e-84b7-c13a67c211da" containerName="revoke-keys"
Nov 02 13:45:03.238590 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.238564    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.254768 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.254742    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:45:03.275107 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.275075    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"]
Nov 02 13:45:03.275268 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.275206    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:45:03.275840 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.275799    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:45:03.275943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.275929    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:45:03.444757 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.444644    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.445104 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.445079    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxd9s\" (UniqueName: \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.445313 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.445293    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.551229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.550361    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.551229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.550442    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxd9s\" (UniqueName: \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.551229 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.550485    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.552769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.552741    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.569983 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.569927    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.578570 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.578519    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxd9s\" (UniqueName: \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\") pod \"maas-api-revoke-sub-3e31a506-rw55s\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:03.859882 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:03.859766    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:04.515321 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:04.515289    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"]
Nov 02 13:45:04.519328 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:45:04.519300    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b30ecd_8b0e_4e69_ac9d_829234449f50.slice/crio-8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22 WatchSource:0}: Error finding container 8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22: Status 404 returned error can't find the container with id 8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22
Nov 02 13:45:05.105902 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:05.105862    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s" event={"ID":"e0b30ecd-8b0e-4e69-ac9d-829234449f50","Type":"ContainerStarted","Data":"677530f83de6e677e67d8fbe1cb7318c884600c7cef0d38c084de0cbbcb724ee"}
Nov 02 13:45:05.106084 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:05.105934    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s" event={"ID":"e0b30ecd-8b0e-4e69-ac9d-829234449f50","Type":"ContainerStarted","Data":"8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22"}
Nov 02 13:45:05.166562 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:05.166501    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s" podStartSLOduration=2.166485002 podStartE2EDuration="2.166485002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:05.152741224 +0000 UTC m=+4702.801795182" watchObservedRunningTime="2026-11-02 13:45:05.166485002 +0000 UTC m=+4702.815538960"
Nov 02 13:45:06.113233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:06.113184    4928 generic.go:356] "Generic (PLEG): container finished" podID="e0b30ecd-8b0e-4e69-ac9d-829234449f50" containerID="677530f83de6e677e67d8fbe1cb7318c884600c7cef0d38c084de0cbbcb724ee" exitCode=0
Nov 02 13:45:06.113648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:06.113311    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s" event={"ID":"e0b30ecd-8b0e-4e69-ac9d-829234449f50","Type":"ContainerDied","Data":"677530f83de6e677e67d8fbe1cb7318c884600c7cef0d38c084de0cbbcb724ee"}
Nov 02 13:45:07.261389 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.261359    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:07.338131 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.338092    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\") pod \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") "
Nov 02 13:45:07.338332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.338141    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\" (UniqueName: \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\") pod \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") "
Nov 02 13:45:07.338332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.338209    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\") pod \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\" (UID: \"e0b30ecd-8b0e-4e69-ac9d-829234449f50\") "
Nov 02 13:45:07.338549 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.338523    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca" pod "e0b30ecd-8b0e-4e69-ac9d-829234449f50" (UID: "e0b30ecd-8b0e-4e69-ac9d-829234449f50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:07.338666 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.338652    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:45:07.340669 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.340631    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca" pod "e0b30ecd-8b0e-4e69-ac9d-829234449f50" (UID: "e0b30ecd-8b0e-4e69-ac9d-829234449f50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:07.340790 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.340733    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s" pod "e0b30ecd-8b0e-4e69-ac9d-829234449f50" (UID: "e0b30ecd-8b0e-4e69-ac9d-829234449f50"). InnerVolumeSpecName "kube-api-access-kxd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:07.440036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.439949    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxd9s\" (UniqueName: \"kubernetes.io/projected/e0b30ecd-8b0e-4e69-ac9d-829234449f50-kube-api-access-kxd9s\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:45:07.440036 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:07.439981    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b30ecd-8b0e-4e69-ac9d-829234449f50-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:45:08.126428 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:08.126394    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"
Nov 02 13:45:08.126648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:08.126422    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s" event={"ID":"e0b30ecd-8b0e-4e69-ac9d-829234449f50","Type":"ContainerDied","Data":"8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22"}
Nov 02 13:45:08.126648 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:45:08.126472    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8680a9d24a055c64374f1906c100f2316ef1bf6ae249126f0b906b064b97bb22"
Nov 02 13:45:09.728654 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:45:09.728603    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:45:39.727603 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:45:39.727559    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:46:09.728541 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:46:09.728501    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:46:36.030149 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:36.030111    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"]
Nov 02 13:46:36.043320 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:36.043277    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13cfd66b-92ms4"]
Nov 02 13:46:36.497256 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:36.497223    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b39dff4-7faf-41fa-82ce-6773154ead5a" path="/var/lib/kubelet/pods/1b39dff4-7faf-41fa-82ce-6773154ead5a/volumes"
Nov 02 13:46:39.728585 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:46:39.728544    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:46:43.492491 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:43.492460    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:46:43.508377 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:43.508349    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:46:51.031889 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:51.031850    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"]
Nov 02 13:46:51.047649 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:51.047609    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fa587-9xwtl"]
Nov 02 13:46:52.500796 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:46:52.500757    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a641d4-2e20-4162-9643-181ce2880d64" path="/var/lib/kubelet/pods/78a641d4-2e20-4162-9643-181ce2880d64/volumes"
Nov 02 13:47:09.728146 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:47:09.728098    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:47:39.730729 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:47:39.730682    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:47:45.555787 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:47:45.555751    4928 scope.go:122] "RemoveContainer" containerID="bf5e00d8b1e13b4a2b07b788f6fb0f3f27d9b245030518fb81fa10ad430aef8b"
Nov 02 13:48:09.728269 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:48:09.728229    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:48:39.728867 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:48:39.728795    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:48:45.578592 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:48:45.578556    4928 scope.go:122] "RemoveContainer" containerID="2e4796d33845f6eaebb77e76bbc22949f68aeb63006a84f361d9f20757e5ad4a"
Nov 02 13:49:09.728265 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:49:09.728219    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:49:39.729247 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:49:39.729196    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:50:07.031253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:07.031167    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"]
Nov 02 13:50:07.043523 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:07.043490    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rw55s"]
Nov 02 13:50:08.497656 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:08.497619    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b30ecd-8b0e-4e69-ac9d-829234449f50" path="/var/lib/kubelet/pods/e0b30ecd-8b0e-4e69-ac9d-829234449f50/volumes"
Nov 02 13:50:09.728259 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:50:09.728217    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:50:39.729536 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:50:39.729497    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:50:45.608599 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:45.608566    4928 scope.go:122] "RemoveContainer" containerID="39299a6ccf206d487fde5f6970083035f82072bb6fe3e4d7b93bee2a178f9783"
Nov 02 13:50:58.833015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:58.832964    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:50:58.833547 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:50:58.833047    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:09.728353 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:51:09.728314    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:51:28.869901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:51:28.869635    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:51:28.869901 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:51:28.869781    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:39.728708 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:51:39.728668    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:51:43.531875 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:51:43.531845    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:51:43.548489 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:51:43.548465    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:51:45.637865 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:51:45.637809    4928 scope.go:122] "RemoveContainer" containerID="677530f83de6e677e67d8fbe1cb7318c884600c7cef0d38c084de0cbbcb724ee"
Nov 02 13:52:09.727388 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:52:09.727341    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:52:39.729064 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:52:39.729019    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:53:07.873838 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.873728    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"]
Nov 02 13:53:07.876295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.874284    4928 cpu_manager.go:414] "containerMap: removing container" podUID="e0b30ecd-8b0e-4e69-ac9d-829234449f50" containerName="revoke-keys"
Nov 02 13:53:07.876295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.874303    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b30ecd-8b0e-4e69-ac9d-829234449f50" containerName="revoke-keys"
Nov 02 13:53:07.876295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.874390    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b30ecd-8b0e-4e69-ac9d-829234449f50" containerName="revoke-keys"
Nov 02 13:53:07.877401 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.877379    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:07.896660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.896608    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:53:07.896660 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.896625    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:53:07.910145 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.910118    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:53:07.910332 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.910308    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:53:07.912579 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.912551    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"]
Nov 02 13:53:07.945650 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.945624    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdtv\" (UniqueName: \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:07.945774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.945666    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:07.945774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:07.945699    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.047479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.047444    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdtv\" (UniqueName: \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.047479 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.047486    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.047731 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.047585    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.048224 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.048202    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.049943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.049925    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.072342 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.072312    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdtv\" (UniqueName: \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\") pod \"maas-api-revoke-sub-bb8dc3c5-pktvz\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.190152 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.190063    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:08.497062 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:53:08.497026    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f294e1_73bf_473a_bbe8_387745ec279b.slice/crio-83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2 WatchSource:0}: Error finding container 83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2: Status 404 returned error can't find the container with id 83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2
Nov 02 13:53:08.498003 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:08.497976    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"]
Nov 02 13:53:09.295465 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:09.295414    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz" event={"ID":"60f294e1-73bf-473a-bbe8-387745ec279b","Type":"ContainerStarted","Data":"d2ce6dbb3a1028dfa9b2d089a70f5b31cb822c87459a9b40e6017b51b05e92d8"}
Nov 02 13:53:09.296037 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:09.295515    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz" event={"ID":"60f294e1-73bf-473a-bbe8-387745ec279b","Type":"ContainerStarted","Data":"83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2"}
Nov 02 13:53:09.339351 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:09.339288    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz" podStartSLOduration=2.33927264 podStartE2EDuration="2.33927264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:09.338289351 +0000 UTC m=+5186.987343308" watchObservedRunningTime="2026-11-02 13:53:09.33927264 +0000 UTC m=+5186.988326593"
Nov 02 13:53:09.728516 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:53:09.728425    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:53:10.303343 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:10.303310    4928 generic.go:356] "Generic (PLEG): container finished" podID="60f294e1-73bf-473a-bbe8-387745ec279b" containerID="d2ce6dbb3a1028dfa9b2d089a70f5b31cb822c87459a9b40e6017b51b05e92d8" exitCode=0
Nov 02 13:53:10.303773 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:10.303374    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz" event={"ID":"60f294e1-73bf-473a-bbe8-387745ec279b","Type":"ContainerDied","Data":"d2ce6dbb3a1028dfa9b2d089a70f5b31cb822c87459a9b40e6017b51b05e92d8"}
Nov 02 13:53:11.437668 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.437637    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:11.586393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.586308    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\") pod \"60f294e1-73bf-473a-bbe8-387745ec279b\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") "
Nov 02 13:53:11.586393 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.586351    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\") pod \"60f294e1-73bf-473a-bbe8-387745ec279b\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") "
Nov 02 13:53:11.586607 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.586442    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\" (UniqueName: \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\") pod \"60f294e1-73bf-473a-bbe8-387745ec279b\" (UID: \"60f294e1-73bf-473a-bbe8-387745ec279b\") "
Nov 02 13:53:11.586774 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.586749    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca" pod "60f294e1-73bf-473a-bbe8-387745ec279b" (UID: "60f294e1-73bf-473a-bbe8-387745ec279b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:53:11.588462 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.588434    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca" pod "60f294e1-73bf-473a-bbe8-387745ec279b" (UID: "60f294e1-73bf-473a-bbe8-387745ec279b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:11.588555 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.588526    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv" pod "60f294e1-73bf-473a-bbe8-387745ec279b" (UID: "60f294e1-73bf-473a-bbe8-387745ec279b"). InnerVolumeSpecName "kube-api-access-qbdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:11.687943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.687906    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbdtv\" (UniqueName: \"kubernetes.io/projected/60f294e1-73bf-473a-bbe8-387745ec279b-kube-api-access-qbdtv\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:53:11.687943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.687936    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:53:11.687943 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:11.687948    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f294e1-73bf-473a-bbe8-387745ec279b-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:53:12.314655 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:12.314612    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz" event={"ID":"60f294e1-73bf-473a-bbe8-387745ec279b","Type":"ContainerDied","Data":"83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2"}
Nov 02 13:53:12.314655 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:12.314659    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a598a59e86218693d2c6177039cf040ba2ccb276a552fd23cf47418e13c6c2"
Nov 02 13:53:12.314920 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:53:12.314621    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"
Nov 02 13:53:39.728483 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:53:39.728443    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:54:07.637253 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.637209    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"]
Nov 02 13:54:07.638009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.637984    4928 cpu_manager.go:414] "containerMap: removing container" podUID="60f294e1-73bf-473a-bbe8-387745ec279b" containerName="revoke-keys"
Nov 02 13:54:07.638111 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.638011    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f294e1-73bf-473a-bbe8-387745ec279b" containerName="revoke-keys"
Nov 02 13:54:07.638175 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.638151    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f294e1-73bf-473a-bbe8-387745ec279b" containerName="revoke-keys"
Nov 02 13:54:07.666538 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.666497    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"]
Nov 02 13:54:07.666746 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.666631    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.682969 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.682940    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:54:07.696505 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.696478    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:54:07.696626 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.696608    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:54:07.696724 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.696705    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:54:07.758119 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.758087    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bv72\" (UniqueName: \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.758295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.758126    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.758295 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.758149    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.859645 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.859603    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bv72\" (UniqueName: \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.859645 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.859647    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.859927 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.859670    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.860347 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.860322    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.862147 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.862118    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.884456 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.884426    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bv72\" (UniqueName: \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\") pod \"maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:07.983210 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:07.983124    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:08.287955 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:08.287930    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"]
Nov 02 13:54:08.289320 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:54:08.289291    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d935c1_5c11_49c5_a351_342a9a539f41.slice/crio-c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a WatchSource:0}: Error finding container c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a: Status 404 returned error can't find the container with id c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a
Nov 02 13:54:08.675128 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:08.675091    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m" event={"ID":"55d935c1-5c11-49c5-a351-342a9a539f41","Type":"ContainerStarted","Data":"b0d659e9e6c89d672f3fbcd859aa7005f11558f178b05b41c6881d45a6befb1a"}
Nov 02 13:54:08.675561 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:08.675144    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m" event={"ID":"55d935c1-5c11-49c5-a351-342a9a539f41","Type":"ContainerStarted","Data":"c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a"}
Nov 02 13:54:08.719892 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:08.719810    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m" podStartSLOduration=1.71979552 podStartE2EDuration="1.71979552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:08.717956698 +0000 UTC m=+5246.367010652" watchObservedRunningTime="2026-11-02 13:54:08.71979552 +0000 UTC m=+5246.368849478"
Nov 02 13:54:09.682548 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:09.682508    4928 generic.go:356] "Generic (PLEG): container finished" podID="55d935c1-5c11-49c5-a351-342a9a539f41" containerID="b0d659e9e6c89d672f3fbcd859aa7005f11558f178b05b41c6881d45a6befb1a" exitCode=0
Nov 02 13:54:09.683027 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:09.682596    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m" event={"ID":"55d935c1-5c11-49c5-a351-342a9a539f41","Type":"ContainerDied","Data":"b0d659e9e6c89d672f3fbcd859aa7005f11558f178b05b41c6881d45a6befb1a"}
Nov 02 13:54:09.729540 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:54:09.729497    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:54:10.826519 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.826486    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:10.901623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.901578    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\") pod \"55d935c1-5c11-49c5-a351-342a9a539f41\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") "
Nov 02 13:54:10.901853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.901780    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\") pod \"55d935c1-5c11-49c5-a351-342a9a539f41\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") "
Nov 02 13:54:10.901853 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.901841    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\" (UniqueName: \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\") pod \"55d935c1-5c11-49c5-a351-342a9a539f41\" (UID: \"55d935c1-5c11-49c5-a351-342a9a539f41\") "
Nov 02 13:54:10.902026 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.901999    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca" pod "55d935c1-5c11-49c5-a351-342a9a539f41" (UID: "55d935c1-5c11-49c5-a351-342a9a539f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:10.902586 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.902559    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-openshift-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:54:10.904013 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.903989    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca" pod "55d935c1-5c11-49c5-a351-342a9a539f41" (UID: "55d935c1-5c11-49c5-a351-342a9a539f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:10.904143 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:10.904132    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72" pod "55d935c1-5c11-49c5-a351-342a9a539f41" (UID: "55d935c1-5c11-49c5-a351-342a9a539f41"). InnerVolumeSpecName "kube-api-access-5bv72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:11.003334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:11.003300    4928 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d935c1-5c11-49c5-a351-342a9a539f41-maas-api-xks-ca\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:54:11.003334 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:11.003332    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bv72\" (UniqueName: \"kubernetes.io/projected/55d935c1-5c11-49c5-a351-342a9a539f41-kube-api-access-5bv72\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:54:11.695062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:11.695017    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m" event={"ID":"55d935c1-5c11-49c5-a351-342a9a539f41","Type":"ContainerDied","Data":"c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a"}
Nov 02 13:54:11.695062 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:11.695050    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"
Nov 02 13:54:11.695298 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:11.695072    4928 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c715ab87031316eac21e6b2a5621fb593b53a7e19d6b34bf3fd9134b13586c3a"
Nov 02 13:54:21.026357 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:21.026317    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"]
Nov 02 13:54:21.039504 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:21.039463    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-593fe3-ptq6m"]
Nov 02 13:54:22.497072 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:22.497034    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d935c1-5c11-49c5-a351-342a9a539f41" path="/var/lib/kubelet/pods/55d935c1-5c11-49c5-a351-342a9a539f41/volumes"
Nov 02 13:54:39.729229 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:54:39.729186    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:54:40.466582 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:40.466541    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 13:54:40.466799 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:40.466741    4928 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" podUID="3a43c64c-0e99-4fb3-a131-042645c331f8" containerName="manager" containerID="cri-o://f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646" gracePeriod=10
Nov 02 13:54:41.723196 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.723168    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 13:54:41.834363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.834284    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\" (UniqueName: \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\") pod \"3a43c64c-0e99-4fb3-a131-042645c331f8\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") "
Nov 02 13:54:41.834363 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.834346    4928 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\") pod \"3a43c64c-0e99-4fb3-a131-042645c331f8\" (UID: \"3a43c64c-0e99-4fb3-a131-042645c331f8\") "
Nov 02 13:54:41.838303 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.835300    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume" pod "3a43c64c-0e99-4fb3-a131-042645c331f8" (UID: "3a43c64c-0e99-4fb3-a131-042645c331f8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:41.844992 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.844958    4928 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t" pod "3a43c64c-0e99-4fb3-a131-042645c331f8" (UID: "3a43c64c-0e99-4fb3-a131-042645c331f8"). InnerVolumeSpecName "kube-api-access-wjl5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:41.903488 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.903446    4928 generic.go:356] "Generic (PLEG): container finished" podID="3a43c64c-0e99-4fb3-a131-042645c331f8" containerID="f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646" exitCode=0
Nov 02 13:54:41.903665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.903490    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" event={"ID":"3a43c64c-0e99-4fb3-a131-042645c331f8","Type":"ContainerDied","Data":"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"}
Nov 02 13:54:41.903665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.903547    4928 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"
Nov 02 13:54:41.903665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.903593    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x" event={"ID":"3a43c64c-0e99-4fb3-a131-042645c331f8","Type":"ContainerDied","Data":"e3ba84f0d858affb675b710654b0d5ba49161bb8e7ff5a313feb58c795b78cb0"}
Nov 02 13:54:41.903665 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.903617    4928 scope.go:122] "RemoveContainer" containerID="f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"
Nov 02 13:54:41.917159 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.917139    4928 scope.go:122] "RemoveContainer" containerID="f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"
Nov 02 13:54:41.917518 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:54:41.917496    4928 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646\": container with ID starting with f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646 not found: ID does not exist" containerID="f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"
Nov 02 13:54:41.917623 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.917524    4928 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646"} err="failed to get container status \"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646\": rpc error: code = NotFound desc = could not find container \"f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646\": container with ID starting with f31b474edc46927760750613c0f9a0c4d89bb0abb8a0466c548b2750f414b646 not found: ID does not exist"
Nov 02 13:54:41.936021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.935996    4928 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjl5t\" (UniqueName: \"kubernetes.io/projected/3a43c64c-0e99-4fb3-a131-042645c331f8-kube-api-access-wjl5t\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:54:41.936021 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.936019    4928 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a43c64c-0e99-4fb3-a131-042645c331f8-extensions-socket-volume\") on node \"ip-10-0-135-160.ec2.internal\" DevicePath \"\""
Nov 02 13:54:41.960255 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.960229    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 13:54:41.964134 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:41.964108    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpm5x"]
Nov 02 13:54:42.496023 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:54:42.495979    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a43c64c-0e99-4fb3-a131-042645c331f8" path="/var/lib/kubelet/pods/3a43c64c-0e99-4fb3-a131-042645c331f8/volumes"
Nov 02 13:55:09.728247 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:55:09.728204    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:55:39.727038 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:55:39.727001    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:55:43.594803 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.594765    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"]
Nov 02 13:55:43.595233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595180    4928 cpu_manager.go:414] "containerMap: removing container" podUID="55d935c1-5c11-49c5-a351-342a9a539f41" containerName="revoke-keys"
Nov 02 13:55:43.595233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595192    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d935c1-5c11-49c5-a351-342a9a539f41" containerName="revoke-keys"
Nov 02 13:55:43.595233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595217    4928 cpu_manager.go:414] "containerMap: removing container" podUID="3a43c64c-0e99-4fb3-a131-042645c331f8" containerName="manager"
Nov 02 13:55:43.595233 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595222    4928 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a43c64c-0e99-4fb3-a131-042645c331f8" containerName="manager"
Nov 02 13:55:43.595387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595284    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a43c64c-0e99-4fb3-a131-042645c331f8" containerName="manager"
Nov 02 13:55:43.595387 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.595297    4928 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d935c1-5c11-49c5-a351-342a9a539f41" containerName="revoke-keys"
Nov 02 13:55:43.598372 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.598355    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.628769 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.628743    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nt9q8\""
Nov 02 13:55:43.631159 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.631136    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"]
Nov 02 13:55:43.641458 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.641432    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/683eff4d-e797-4a7f-a4cc-7efdbf51896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.641592 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.641563    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn64\" (UniqueName: \"kubernetes.io/projected/683eff4d-e797-4a7f-a4cc-7efdbf51896d-kube-api-access-8vn64\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.742909 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.742879    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn64\" (UniqueName: \"kubernetes.io/projected/683eff4d-e797-4a7f-a4cc-7efdbf51896d-kube-api-access-8vn64\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.743125 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.742952    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/683eff4d-e797-4a7f-a4cc-7efdbf51896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.743427 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.743400    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/683eff4d-e797-4a7f-a4cc-7efdbf51896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.767897 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.767867    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn64\" (UniqueName: \"kubernetes.io/projected/683eff4d-e797-4a7f-a4cc-7efdbf51896d-kube-api-access-8vn64\") pod \"kuadrant-operator-controller-manager-678d87766f-5nrjn\" (UID: \"683eff4d-e797-4a7f-a4cc-7efdbf51896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:43.911132 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:43.911027    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:44.248415 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:44.248389    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"]
Nov 02 13:55:44.257982 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:55:44.257949    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683eff4d_e797_4a7f_a4cc_7efdbf51896d.slice/crio-f8c1badf76ea9aa6fbe6eabb36d872cd9b5f3b856f30217ef6a80074b270377f WatchSource:0}: Error finding container f8c1badf76ea9aa6fbe6eabb36d872cd9b5f3b856f30217ef6a80074b270377f: Status 404 returned error can't find the container with id f8c1badf76ea9aa6fbe6eabb36d872cd9b5f3b856f30217ef6a80074b270377f
Nov 02 13:55:45.245735 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:45.245698    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn" event={"ID":"683eff4d-e797-4a7f-a4cc-7efdbf51896d","Type":"ContainerStarted","Data":"9b861b6c2aef5abaeff8bdcd0eb1581c29135bbbd126e793ba0c3ff9c3bedf3d"}
Nov 02 13:55:45.246209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:45.245749    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn" event={"ID":"683eff4d-e797-4a7f-a4cc-7efdbf51896d","Type":"ContainerStarted","Data":"f8c1badf76ea9aa6fbe6eabb36d872cd9b5f3b856f30217ef6a80074b270377f"}
Nov 02 13:55:45.246209 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:45.245813    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:55:45.319883 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:45.319813    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn" podStartSLOduration=2.319797213 podStartE2EDuration="2.319797213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:45.318334593 +0000 UTC m=+5342.967388575" watchObservedRunningTime="2026-11-02 13:55:45.319797213 +0000 UTC m=+5342.968851170"
Nov 02 13:55:56.253401 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:55:56.253317    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5nrjn"
Nov 02 13:56:09.728184 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:56:09.728141    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:56:39.727688 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:56:39.727651    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:56:43.570133 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.570103    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:56:43.596301 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.596276    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 13:56:43.784791 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.784754    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"]
Nov 02 13:56:43.788430 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.788411    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.818369 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.818314    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:43.818369 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.818318    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:43.818680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.818571    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:56:43.818680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.818579    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 13:56:43.818680 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.818666    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:56:43.821706 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.821647    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"]
Nov 02 13:56:43.897786 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.897751    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftqk\" (UniqueName: \"kubernetes.io/projected/5736a03c-867f-4ea1-bb59-49d58af8c4bd-kube-api-access-cftqk\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.898019 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.897810    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-metrics-certs\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.898019 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.897946    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-maas-api-tls\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.898019 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.897994    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736a03c-867f-4ea1-bb59-49d58af8c4bd-openshift-service-ca\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.999035 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.998995    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftqk\" (UniqueName: \"kubernetes.io/projected/5736a03c-867f-4ea1-bb59-49d58af8c4bd-kube-api-access-cftqk\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.999223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.999067    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-metrics-certs\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.999223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.999090    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-maas-api-tls\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.999223 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.999121    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736a03c-867f-4ea1-bb59-49d58af8c4bd-openshift-service-ca\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:43.999895 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:43.999869    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736a03c-867f-4ea1-bb59-49d58af8c4bd-openshift-service-ca\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:44.001628 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.001601    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-maas-api-tls\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:44.001723 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.001657    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736a03c-867f-4ea1-bb59-49d58af8c4bd-metrics-certs\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:44.023741 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.023710    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftqk\" (UniqueName: \"kubernetes.io/projected/5736a03c-867f-4ea1-bb59-49d58af8c4bd-kube-api-access-cftqk\") pod \"maas-api-756c48cb77-q6fb5\" (UID: \"5736a03c-867f-4ea1-bb59-49d58af8c4bd\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:44.101673 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.101594    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:44.409429 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.409403    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"]
Nov 02 13:56:44.411913 ip-10-0-135-160 kubenswrapper[4928]: W1102 13:56:44.411877    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5736a03c_867f_4ea1_bb59_49d58af8c4bd.slice/crio-8e3eb8a26b06ef5e9fb95ee394d927c05da9cdc7132e353105ceb6441d1fd073 WatchSource:0}: Error finding container 8e3eb8a26b06ef5e9fb95ee394d927c05da9cdc7132e353105ceb6441d1fd073: Status 404 returned error can't find the container with id 8e3eb8a26b06ef5e9fb95ee394d927c05da9cdc7132e353105ceb6441d1fd073
Nov 02 13:56:44.413397 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.413378    4928 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:56:44.621186 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:44.621149    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5" event={"ID":"5736a03c-867f-4ea1-bb59-49d58af8c4bd","Type":"ContainerStarted","Data":"8e3eb8a26b06ef5e9fb95ee394d927c05da9cdc7132e353105ceb6441d1fd073"}
Nov 02 13:56:45.627581 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:45.627546    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5" event={"ID":"5736a03c-867f-4ea1-bb59-49d58af8c4bd","Type":"ContainerStarted","Data":"c46da8bb9c36c84d449e7fd104dc3bb38c537b39de26f607e3a5592e999ef8a7"}
Nov 02 13:56:45.628014 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:45.627606    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:56:45.672376 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:45.672306    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5" podStartSLOduration=2.32759158 podStartE2EDuration="2.672285827s" totalImagesPullingTime="344.694247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:45.671695534 +0000 UTC m=+5403.320749492" watchObservedRunningTime="2026-11-02 13:56:45.672285827 +0000 UTC m=+5403.321339786"
Nov 02 13:56:51.664015 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:56:51.663981    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-756c48cb77-q6fb5"
Nov 02 13:57:09.727993 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:57:09.727952    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:57:39.728664 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:57:39.728619    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:58:09.728407 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:58:09.728370    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:58:12.034012 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:12.033979    4928 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"]
Nov 02 13:58:12.048888 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:12.048854    4928 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8dc3c5-pktvz"]
Nov 02 13:58:12.503783 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:12.503735    4928 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f294e1-73bf-473a-bbe8-387745ec279b" path="/var/lib/kubelet/pods/60f294e1-73bf-473a-bbe8-387745ec279b/volumes"
Nov 02 13:58:28.862054 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:28.861967    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:28.862768 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:28.862712    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:29.024009 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:29.023967    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5g6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:29.024241 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:58:29.024042    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5g6c" podUID="5fa67824-357b-4f1b-941d-4c70f5f14994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:39.728517 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:58:39.728477    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:59:09.728990 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:59:09.728900    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:59:39.762115 ip-10-0-135-160 kubenswrapper[4928]: E1102 13:59:39.762034    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 13:59:45.814346 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:59:45.814309    4928 scope.go:122] "RemoveContainer" containerID="d2ce6dbb3a1028dfa9b2d089a70f5b31cb822c87459a9b40e6017b51b05e92d8"
Nov 02 13:59:58.856986 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:59:58.856940    4928 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f7fjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:58.857770 ip-10-0-135-160 kubenswrapper[4928]: I1102 13:59:58.857728    4928 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f7fjr" podUID="f215c0d5-262f-4c8b-84ef-5496f9a8944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:09.728099 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:00:09.728059    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:00:39.787446 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:00:39.787368    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:00:45.838282 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:00:45.838246    4928 scope.go:122] "RemoveContainer" containerID="b0d659e9e6c89d672f3fbcd859aa7005f11558f178b05b41c6881d45a6befb1a"
Nov 02 14:01:09.734813 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:01:09.734762    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:01:39.732012 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:01:39.731969    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:01:43.616429 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:01:43.616393    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 14:01:43.635563 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:01:43.635539    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 14:01:53.429345 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:01:53.429303    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-756c48cb77-q6fb5_5736a03c-867f-4ea1-bb59-49d58af8c4bd/maas-api/0.log"
Nov 02 14:01:59.458057 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:01:59.458022    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b88v5_fbd6770d-95d5-48fb-8fd2-c17cf1672c78/manager/0.log"
Nov 02 14:01:59.664450 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:01:59.664418    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cxwbd_69b89c26-ba10-4856-8e9c-d13589d2ac73/manager/0.log"
Nov 02 14:02:00.088266 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:00.088227    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5nrjn_683eff4d-e797-4a7f-a4cc-7efdbf51896d/manager/0.log"
Nov 02 14:02:00.548448 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:00.548414    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-v9xwk_675b6f92-53c6-4559-9391-218bfe57c01f/manager/0.log"
Nov 02 14:02:01.523396 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:01.523363    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fnlqcc_69bee30f-e272-4cf6-8b8c-983855b3107c/istio-proxy/0.log"
Nov 02 14:02:01.719334 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:01.719303    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cznj4_a2cc2c54-2032-4ccf-a958-af2200e3d477/discovery/0.log"
Nov 02 14:02:02.526157 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:02.526127    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-4g2zz_ce79f7b4-4d9f-461c-bac8-5ff7576cf72f/istio-proxy/0.log"
Nov 02 14:02:03.134167 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:03.134134    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f5g6c_5fa67824-357b-4f1b-941d-4c70f5f14994/istio-proxy/0.log"
Nov 02 14:02:03.344533 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:03.344504    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f7fjr_f215c0d5-262f-4c8b-84ef-5496f9a8944f/istio-proxy/0.log"
Nov 02 14:02:04.365799 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:04.365766    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth_fea80c1c-f77d-4271-a069-73021aaae094/istio-proxy/0.log"
Nov 02 14:02:04.906329 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:04.906298    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d86d55f7-vpzjg_82d00d41-1c4a-450f-9a27-8571dc21bf21/router/0.log"
Nov 02 14:02:06.255780 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:06.255745    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hrm6f_e9fdc759-e3a5-4368-88d9-eb9f1e31a7a7/main/0.log"
Nov 02 14:02:06.639677 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:06.639597    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f9vnt_c51b659b-07c3-40f5-ad34-c8fc405145d5/main/0.log"
Nov 02 14:02:07.030987 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:07.030955    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kxh95_39a8e1f5-a620-4814-8913-5ed14af5ccdd/main/0.log"
Nov 02 14:02:07.233075 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:07.233043    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-j9lq5_d3e5f98e-cab4-4a6b-8884-efda80d420ec/main/0.log"
Nov 02 14:02:09.728450 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:02:09.728408    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:02:22.596185 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:22.596149    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qqgcj_c5b28cf8-a271-41c8-a765-811b45bfeeb0/global-pull-secret-syncer/0.log"
Nov 02 14:02:23.003910 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:23.003880    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9nkss_0e8b0b9d-80b8-460e-98d3-189c240cd20d/connection-checker/0.log"
Nov 02 14:02:23.144575 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:23.144543    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hzn62_ce9a1148-d94a-47ad-8b2d-147af29a5797/connection-checker/0.log"
Nov 02 14:02:23.297741 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:23.297656    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-m5bqj_04856dfa-3c20-45b0-bc5f-f68896f87c51/connection-checker/0.log"
Nov 02 14:02:23.858227 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:23.858194    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dwphp_69559f09-3ce2-435a-a738-1d6892629b7d/konnectivity-agent/0.log"
Nov 02 14:02:24.286364 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:24.286330    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-160.ec2.internal_fc7bed3dd43ef144d6694274c9c4b683/haproxy/0.log"
Nov 02 14:02:30.926819 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:30.926788    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b88v5_fbd6770d-95d5-48fb-8fd2-c17cf1672c78/manager/0.log"
Nov 02 14:02:31.070601 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:31.070562    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cxwbd_69b89c26-ba10-4856-8e9c-d13589d2ac73/manager/0.log"
Nov 02 14:02:31.365227 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:31.365192    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5nrjn_683eff4d-e797-4a7f-a4cc-7efdbf51896d/manager/0.log"
Nov 02 14:02:31.686705 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:31.686615    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-v9xwk_675b6f92-53c6-4559-9391-218bfe57c01f/manager/0.log"
Nov 02 14:02:32.328161 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.328120    4928 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"]
Nov 02 14:02:32.331886 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.331867    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.357673 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.357649    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"]
Nov 02 14:02:32.367345 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.367323    4928 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trmxf\"/\"default-dockercfg-p5wzj\""
Nov 02 14:02:32.432440 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.432402    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"openshift-service-ca.crt\""
Nov 02 14:02:32.432440 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.432428    4928 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"kube-root-ca.crt\""
Nov 02 14:02:32.441978 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.441948    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdfs\" (UniqueName: \"kubernetes.io/projected/70c22cbc-d16e-4177-8d14-61e2e12cd562-kube-api-access-8rdfs\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.442123 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.441985    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-proc\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.442123 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.442070    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-podres\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.442123 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.442112    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-sys\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.442267 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.442142    4928 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-lib-modules\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.543821 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.543781    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-lib-modules\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.543930    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdfs\" (UniqueName: \"kubernetes.io/projected/70c22cbc-d16e-4177-8d14-61e2e12cd562-kube-api-access-8rdfs\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.543943    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-lib-modules\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.543952    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-proc\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.544001    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-proc\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.544013    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-podres\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.544048    4928 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-sys\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544054 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.544053    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-podres\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.544305 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.544093    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70c22cbc-d16e-4177-8d14-61e2e12cd562-sys\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.569422 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.569387    4928 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdfs\" (UniqueName: \"kubernetes.io/projected/70c22cbc-d16e-4177-8d14-61e2e12cd562-kube-api-access-8rdfs\") pod \"perf-node-gather-daemonset-547n6\" (UID: \"70c22cbc-d16e-4177-8d14-61e2e12cd562\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.645053 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.644966    4928 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:32.950072 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:32.950047    4928 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"]
Nov 02 14:02:32.952447 ip-10-0-135-160 kubenswrapper[4928]: W1102 14:02:32.952417    4928 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod70c22cbc_d16e_4177_8d14_61e2e12cd562.slice/crio-02ca07ed2364cb0b368ec62d4e45865cbe0d87264b1166da01f676864e9ec864 WatchSource:0}: Error finding container 02ca07ed2364cb0b368ec62d4e45865cbe0d87264b1166da01f676864e9ec864: Status 404 returned error can't find the container with id 02ca07ed2364cb0b368ec62d4e45865cbe0d87264b1166da01f676864e9ec864
Nov 02 14:02:33.133549 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:33.133510    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6" event={"ID":"70c22cbc-d16e-4177-8d14-61e2e12cd562","Type":"ContainerStarted","Data":"6a5515f608361ff730bf4586fd4e7950f185a47cee042bd543baaca90661e662"}
Nov 02 14:02:33.133757 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:33.133566    4928 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6" event={"ID":"70c22cbc-d16e-4177-8d14-61e2e12cd562","Type":"ContainerStarted","Data":"02ca07ed2364cb0b368ec62d4e45865cbe0d87264b1166da01f676864e9ec864"}
Nov 02 14:02:33.133757 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:33.133638    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:33.174762 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:33.174704    4928 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6" podStartSLOduration=1.174690257 podStartE2EDuration="1.174690257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:33.173149439 +0000 UTC m=+5750.822203398" watchObservedRunningTime="2026-11-02 14:02:33.174690257 +0000 UTC m=+5750.823744214"
Nov 02 14:02:33.893643 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:33.893611    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-d48zc_7cf5eb1e-e80f-40d5-9367-4af0c929723f/cluster-monitoring-operator/0.log"
Nov 02 14:02:34.620833 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:34.620785    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwcrl_8e1addee-ee5f-456d-be31-59caef398724/node-exporter/0.log"
Nov 02 14:02:34.744696 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:34.744670    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwcrl_8e1addee-ee5f-456d-be31-59caef398724/kube-rbac-proxy/0.log"
Nov 02 14:02:34.864969 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:34.864933    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwcrl_8e1addee-ee5f-456d-be31-59caef398724/init-textfile/0.log"
Nov 02 14:02:39.149051 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:39.149023    4928 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-547n6"
Nov 02 14:02:39.728189 ip-10-0-135-160 kubenswrapper[4928]: E1102 14:02:39.728152    4928 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qnhns" containerName="extractor"
Nov 02 14:02:39.808278 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:39.808232    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-8gt88_5e586231-9b95-4114-82c2-602480b1e3c9/networking-console-plugin/0.log"
Nov 02 14:02:40.886063 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:40.886033    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/3.log"
Nov 02 14:02:40.889341 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:40.889319    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-55ghk_8e2649ce-6913-40b6-a47a-cd71b5208dd3/console-operator/4.log"
Nov 02 14:02:43.282024 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:43.281985    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xwbjh_8250eede-bd12-46d3-a569-6a242bbfd566/volume-data-source-validator/0.log"
Nov 02 14:02:45.314868 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:45.314816    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fnlqcc_69bee30f-e272-4cf6-8b8c-983855b3107c/istio-proxy/0.log"
Nov 02 14:02:45.549327 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:45.549294    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2928m_7e3c9141-625c-449d-8b94-3aa76c9a5d04/dns/0.log"
Nov 02 14:02:45.688478 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:45.688368    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2928m_7e3c9141-625c-449d-8b94-3aa76c9a5d04/kube-rbac-proxy/0.log"
Nov 02 14:02:46.365018 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:46.364971    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2c8j4_9939d8b4-1181-463b-a465-13da054067b6/dns-node-resolver/0.log"
Nov 02 14:02:47.887499 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:47.887462    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cznj4_a2cc2c54-2032-4ccf-a958-af2200e3d477/discovery/0.log"
Nov 02 14:02:48.065181 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.065150    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdn4d_a7d37dd0-8676-4b17-81a4-201865d84b89/node-ca/0.log"
Nov 02 14:02:48.150919 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.150817    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-4g2zz_ce79f7b4-4d9f-461c-bac8-5ff7576cf72f/istio-proxy/0.log"
Nov 02 14:02:48.362038 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.362006    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f5g6c_5fa67824-357b-4f1b-941d-4c70f5f14994/istio-proxy/0.log"
Nov 02 14:02:48.434258 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.434166    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f7fjr_f215c0d5-262f-4c8b-84ef-5496f9a8944f/istio-proxy/0.log"
Nov 02 14:02:48.781084 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.781049    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-kxgth_fea80c1c-f77d-4271-a069-73021aaae094/istio-proxy/0.log"
Nov 02 14:02:48.991165 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:48.991132    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d86d55f7-vpzjg_82d00d41-1c4a-450f-9a27-8571dc21bf21/router/0.log"
Nov 02 14:02:50.542613 ip-10-0-135-160 kubenswrapper[4928]: I1102 14:02:50.542578    4928 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fnlqcc_69bee30f-e272-4cf6-8b8c-983855b3107c/istio-proxy/0.log"